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growth. A self-report survey method elicited responses from adult educators 
who currently teach or previously taught adult education courses in United 
States institutions offering graduate degrees in adult education. Of 155 
questionnaires, 78 were returned and 68 were usable. Data were summarized 
using frequencies, percents, means, and standard deviations. Some tests of 
statistical significance were carried out using the chi-square statistic. No 
statistically significant results were found. Respondents agreed that one 
goal of adult education should be to develop students' critical thinking 
skills. Moreover, they believed they are teaching critical thinking, using an 
indirect approach. However, they indicated that adult educators do not have a 
clear idea about what critical thinking is, which suggests that instruction 
has not been addressed systematically. Although no single definition of 
critical thinking emerged, results led to development of a conceptual 
framework of critical thinking for adult educators. Results indicated adult 
educators used experiential and participatory methods when teaching critical 
thinking skills. To evaluate gains in students' critical thinking skills, 
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One of the goals of adult education is to develop students’ abilities to think 
critically, yet there is a paucity of research to guide adult educators in their pursuit of this 
goal. This exploratory study was designed to examine adult educators’ perceptions of 
components related to critical thinking, including definitions of critical thinking, 
instructional methods used for teaching critical thinking, and methods of measuring 
students’ growth in critical thinking. 

A self-report survey method was used to elicit responses from adult educators 
who currently teach or previously taught adult education courses in institutions within the 
United States offering graduate degrees in adult education. Seventy-eight of 1 55 
questionnaires were returned. Sixty-eight questionnaires were usable for a response rate 
of 47%. 

Data were summarized using frequencies, percents, means, and standard 
deviations. Some tests of statistical significance were carried out using the chi-square 
statistic. No statistically significant results were found. 

Respondents agreed one of the goals of adult education should be to develop 
students’ critical thinking skills. Moreover, they believed they are teaching critical 
thinking, using an indirect approach. However, they indicated that adult educators do not 
have a clear idea about what critical thinking is which suggests critical thinking 
instruction has not been addressed systematically. 

Although no single definition of critical thinking emerged, results led to 
development of a conceptual framework of critical thinking for adult educators. The 
framework presents critical thinking as a two-dimensional construct consisting of 
cognitive skills and dispositions. Relative to instructional methods, results indicated 
adult educators used experiential and participatory methods when teaching critical 
thinking skills. To evaluate gains in students’ critical thinking skills, adult educators 
reported using a variety of qualitative measures. 

Further research is needed to develop or adapt a uniform and comprehensive 
definition of critical thinking by adult educators. Additionally, it should be determined if 
critical thinking is perceived differently by adult educators than by educators in other 
disciplines. Also, instructional methods should be examined to determine which are most 
effective in teaching critical thinking skills and dispositions in adult education. This 
would lead to investigation of the validity and reliability of qualitative measures adult 
educators use to measure gains in critical thinking. Finally, longitudinal studies are 
needed to examine if adult learners who show improvement in their ability to think 
critically maintain these gains over time and whether they are able to transfer these skills 
to other areas of their lives such as employment and personal/social. 
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Chapter 1 

INTRODUCTION 

“A democracy cannot survive unthinking citizens. ” 

- Thomas Jefferson 

In the 1980’s, critical thinking skills began receiving increased attention in 
education and business/industry (Facione, 1990; Halpem, 1993; Reboy, 1989; Sternberg, 
1985; Tucker, 1996). Emphasis on critical thinking emerged in response to constant and 
rapid change in every aspect of life, including family, work, community, and society 
(Brookfield, 1987; Glaser & Resnick, 1991; Kerka, 1992). Critical thinking experts 
argued the most effective way to prepare individuals for a productive, full, and satisfying 
life, amidst the constant change, was to help them develop critical thinking skills (Beyer, 
1985; Ennis, 1985; Facione, 1990; Paul, 1982; Penner, 1995; Siegel, 1988; Sternberg, 
1985). 

Halpern (1993) described the increasing complexity of society and concluded that 
learning faster and better may be the only viable responses to a rapidly changing world. 
According to Tomlin (1997), the learning curve began to steepen in the 1980’s and not 
just with more content. With the Information Age, it became nearly impossible to keep 
up with content, and the real task became learning in and of itself. Davis and Botkin 
(1995) described the pace of modem society and noted that no sooner do people adjust to 
a new way of doing things than those things change on them again. For example, the 
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rapid pace of technological change meant education must be updated throughout the 
individual’s working life. Adult educator and futurist Paul Larson (cited in Tomlin, 

1 997) predicted 

the smart people of the future will be those who can learn a new technology very 
very quickly and add value to their job, or who will be able to create a new and 
different job to meet an immediate and new need. In other words, the rules have 
changed and adults must change how and what they learn, (p. 19) 

Lifelong learning has become the norm, and people have to “increase learning power to 

sustain earning power” (Davis & Botkin, 1995, p. 16). 

Sternberg (1985) believed the resurgence of interest in critical thinking resulted 

from a combination of social forces, including: (1) declining scores on tests of scholastic 

aptitude which have called attention to the apparently declining levels of students’ critical 

thinking; (2) a number of national reports which have blamed schools because students 

are not learning to think as well as they should; and (3) the advancement of psycho- 

educational knowledge to the point that programs for teaching critical thinking look more 

promising than ever before. 

Schrag (1992) concurred with Sternberg’s third point, pointing out that “From the 
1950’s until the 1970’s, most educational research was dominated by the behaviorism 
school of psychology which either denied the existence of mental processes or denied 
their accessibility to scientific study” (p. 255). In the 1980’s, however, cognitive 
approaches to psychology began replacing behavioral approaches, and cognitive 
approaches are well suited to the experimental investigation of thinking and problem 
solving. In fact, modem cognitive research has shown that instruction must do more 




3 



than convey factual information. According to Glaser and Resnick (1991), instruction 
must encourage students to ask questions about what they learn, invent new ways of 
solving problems, connect new knowledge to information they already have, and apply 
their knowledge and reasoning skills in new situations. In short, instruction must help 
students develop critical thinking skills. 

Halpem (1993) also supported Sternberg’s analysis of the problem, reporting that 
American students routinely rank below those from other parts of the world in academic 
areas such as scientific knowledge and understanding, mathematical problem solving, and 
general literacy. As a result, economists and politicians worry about the country’s ability 
to remain a world leader in science and technology. To remedy this situation, leaders in 
government and business/industry have demanded that teachers include critical thinking 
in their curriculum. 

Patrick (1986) and Brookfield (1987) described changes in the political realm that 
have occurred in the last two decades of this century and argued that good citizenship 
requires informed participants who have the capability to think critically about public 
issues, candidates for public office, and the decisions of government officials. Penner 
(1995) added that without critical thinking, citizens would believe all politicians’ claims 
and would lose the ability to distinguish good from evil. 

Repeated forecasts indicate the work force of the future will require a much 
greater ability of workers to solve problems on their own (Schrag, 1992). Kerka (1992) 
studied the changing world of work and noted that as the nature of work changes and 
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people work longer, the skills they need are the “capacities to learn continuously through 
thinking and reasoning, problem solving, and decision-making” (p. 2). Thomas (as cited 
in Kerka, 1992) reported that occupations are becoming more reliant on cognitive 
capacities, and the changing work environment requires flexibility and adaptability to 
changing conditions. In support of these views, Feuer and Geber (1988) insisted 
“Today’s organizations need more educated, responsible and flexible work forces 
composed of employees who can think for themselves and direct their own life-long 
learning” (p. 38). 

Another factor contributing to the attention being paid to critical thinking is the 
expanding marketplace. Many authors, including Marsick (cited in Feuer & Geber, 

1988) argued that today’s global marketplace requires employees at many levels of the 
organization to think and act more independently, use judgment, and figure out what the 
nature of the problem is rather than just applying a set of predefined rules. 

Underscoring the pace of change, some researchers (e.g., Toffler cited in Tomlin, 
1997) estimated the United States alone will create 10,000 new jobs a day every day for 
the next ten years. While many of these jobs will be in the service sector, many others 
will be in careers that have not yet been invented. Workers in these new careers do not 
know what or how they will need to learn. 

In response to the demands of a changing society and workplace, leaders at every 
level in education— from elementary/secondary education through adult education— and 
business/industry began calling for the development of critical thinking skills. They 
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argued that successful adaptation to a changing world requires the ability to think 
critically and to synthesize large quantities of new information (National Institute of 
Education, 1984). Howe and Warren (1989) summarized the value placed on critical 
thinking skills: “The ability to think critically is essential if individuals are to live, work, 
and function effectively in our current and changing society” (p. 3). 

The arguments for critical thinking were successful. According to Tucker (1996), 
in the 1980’s and 1990’s several activities promoting critical thinking emerged. Watson 
and Glaser’s (1980) Critical Thinking Appraisal was developed and became a widely 
used tool for assessing the effects of undergraduate education on reasoning skills. The 
publication of A Nation at Risk (TJ.S. Department of Education, 1983) in 1983 resulted in 
increased emphasis on certain core skills, including critical thinking. In 1988 and 1989, 
46 experts, using the Delphi technique, participated in a project (hereafter known as the 
Delphi Project) designed to reach consensus on the essential elements of critical thinking 
for college graduates (Facione, 1990). In 1989, the nation’s governors adopted the 
National Education Goals (U.S. Department of Education, 1993) that identified critical 
thinking as a core skill. By 1990, the Secretary’s Commission to Achieve Necessary 

Skills (SCANS) Report (U.S. Department of Labor, 1990) and the publication of 

■\ 

America’s Choice: High Skills or Low Wages (National Center on Education and the 
Economy, 1990) had focused the nation’s attention on the importance of indispensable 
skills, including critical thinking, in maintaining national productivity and global 
competitiveness. In 1993, the National Education Goals were adopted under Goals 2000 
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(U.S. Department of Education, Office of Educational Research Improvement, 1994) and 

became law. In support of the SCANS Report, the national goals focused on teaching all 

students to become critical thinkers. Professional education programs, nursing education 

among the most notable, made the assessment of critical thinking skills a top priority. 

Human resource development and corporate education departments attempted to find 

better ways to assess and improve thinking and decision-making skills in the workplace. 

In the span of a few years, the critical thinking movement gained momentum in 

elementary and secondary schools, at the undergraduate and graduate levels, in 

professional education programs, and in human resource development programs. In the 

field of adult education, critical thinking assumed an important position as adult 

educators came to recognize with Hallenbeck (cited in Darkenwald & Merriam, 1982) 

that the basic aims of adult education are: 

to maintain an adult population up to the standards of competence in the 
knowledge, wisdom, and skill which society requires; to develop in adults an 
understanding of the serious problems which interrupt the operations and progress 
of their cooperative society and prepare them to participate in the solution of these 
problems; and to provide all adults with opportunities for their highest possible 
development in attitudes, understanding, knowledge, and quality of human 
existence toward the goal of the greater self-fulfillment and realization of each 
individual human being, (pp. 50-51) 

Given the emphasis on critical thinking skills over the past two decades, one 
would expect to see growth in students’ critical thinking skills. Yet a high percentage of 
students are not able to use critical thinking effectively, and business/industry continues 
to report that many employees are not able to think critically in job situations (Howe & 
Warren, 1989). In fact, based on 1992 test results of adult literacy, almost half of all 
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American adults do not perform at the level of literacy considered by the National 
Education Goals Panel to be necessary for competing successfully in a global economy 
and for exercising the rights and responsibilities of citizenship (Gronlund, 1993). 

Statement of the Problem 

“There is no innate tendency to think critically, nor is it easy to acquire. ” 

- R. S. Peters 

Critical thinking skills are not a new topic in education. In fact, the concern to 
teach students how to think and not merely what to think predates the existence of formal 
schooling (Schrag, 1 992). Some (e.g., Schrag) attributed the beginnings of teaching 
critical thinking to Socrates and his dialogue method of discovery. Others (Brookfield, 

1 987; Garrison, 1991) linked the modem conceptualization of critical thinking to John 
Dewey (1933) and his landmark work, How We Think: A Restatement of the Relation of 
Reflective Thinking to the Educative Process, in which Dewey argued that equipping 
students with the ability and desire to solve the problems facing them and the larger 
society was the most important task facing educators. Paul (cited in McPeck, 1990) 
traced the critical thinking movement back to Glaser’s Experiment in the Development of 
Critical Thinking (1941). In the second half of this century, there have been many 
philosophers, psychologists, and educators (e.g., Beyer, 1985; Brookfield, 1987; Ennis, 
1985; Facione, 1990; Garrison, 1991; Halpem, 1993; Kurfiss, 1988; Lipman, 1985; 
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McMillan, 1987; McPeck, 1981; Mezirow, 1981; Norris, 1988; Paul, 1982; Siegel, 1988) 
who have contributed to our understanding of critical thinking. 

Despite both the long-standing and more recent interest in developing students’ 
critical thinking skills, few published studies documented the development of critical 
thinking or demonstrated how to improve it with specific curriculum materials or 
instructional methods (Terenzini, Theophilides, & Lorang, 1984). 

One of the goals of adult education is to develop students' ability to think 
critically (Brookfield, 1987; Candy, 1991; Garrison, 1991; Kummerow cited in Naylor, 
1984; Mezirow, 1981). Lindeman, the father of adult education (cited in Darkenwald & 
Merriam, 1982) emphasized that “Changing individuals in continuous adjustment to 
changing social functions— that is the . . . purpose of adult learning” (p. 52). McPeck 
(1990) agreed with Garrison (1991) that “the development of critical thinking skills 
appears to be more appropriate and teachable in mature learners who have, as a 
precondition, the foundational knowledge and experience” (p. 302). Garrison believed 
that critical thinking is a useful and powerful construct in adult education with which to 
understand learning and knowledge development. Despite the support for critical 
thinking in adult education, Brookfield (1987) reported that “as a dimension of learning, 
critical thinking in adulthood has been largely neglected in the educational literature” (p. 
ix). 

The effort to define critical thinking has received a good deal of attention, yet 
there is no consensus as to its meaning (Garrison, 1991; McMurray, Thompson, & 
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Beisenherz, 1989; Sternberg, 1985; Tucker, 1996). Lipman (1985) requested a set of 
criteria by which critical thinking could be identified, a working definition that could 
guide teachers to encourage their students to think critically and to coach them as they do. 

Patrick (1986) claimed critical thinking has not been taught extensively or 
satisfactorily. Garrison (1991) agreed but suggested that educators are confused about 
how to develop critical thinking in adult learners. Smith (1980) reported that educators 
have advanced a plethora of engaging ideas aimed at improving their students’ critical 
thinking skills but have reported little experimental research to assess the effectiveness of 
their instructional strategies. McMillan (1987) noted there is little evidence that critical 
thinking skills are influenced by specific instructional variables. 

One of the most difficult aspects of critical thinking is how to measure gains in 
students’ critical thinking skills. Despite the importance of this issue, Norris (cited in 
McPeck, 1 990) argued that many of the questions about measuring critical thinking skills 
have been treated by philosophers but without benefit of empirical studies. Tucker 
(1996) criticized extant tests of critical thinking for lack of an underlying empirical 
structure. 

While there is wide agreement that the ability to think critically is one of the 
desirable outcomes of adult education, there is a paucity of research to guide adult 
educators in their pursuit of this goal. Despite the attention given to critical thinking over 
the past two decades, there is no single agreed-upon definition of critical thinking. 
Without a workable definition of what is meant by critical thinking skills, it is difficult 
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for educators to develop appropriate curriculum, effective instructional strategies, or valid 
assessment tools. Moreover, few empirical studies have been reported that demonstrate 
the effectiveness of specific instructional strategies on the development of critical 
thinking skills. Although there is consensus that change in students’ critical thinking 
skills should be documented, there is little empirical evidence to help educators know 
which critical thinking measurements are valid and reliable. 

Purpose of the Study 

t 

The purpose of this study was to examine adult educators’ perceptions of 
components of critical thinking. Specifically, the study attempted to contribute to 
professional understanding of critical thinking by defining critical thinking in the context 
of adult education, documenting instructional methods used by adult educators to teach 
critical thinking, and reporting evaluation techniques used by adult educators to measure 
students’ growth in critical thinking. 

Research Questions 

Five research questions guided this investigation: 

1 . To what extent is critical thinking being taught in adult education courses? 

2. How do adult educators define critical thinking? 

3. Do adult educators have a preference in how critical thinking is taught? 

4. How do adult educators measure gains in students’ critical thinking? 
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5. To what extent do adult educators agree or disagree on topics related to 
teaching critical thinking? 

Significance of the Study 



Given the increasing complexity of our society, the development of critical 
thinking skills is a laudable educational goal (McPeck, 1990). There is widespread 
agreement that the ultimate aim of critical thinking is to improvepeople’s reasoning 
ability about everyday problems and issues. Garrison (1991) reported “the concept of 
critical thinking reflects the increased emphasis that educators place upon cognitive 
processes in knowledge development and problem solving as opposed to simple 
information acquisition” (p. 287). 

In recent years, critical thinking has become a central tenet of the study and 
practice of adult education (Garrison, 1991). As Brookfield (1987) stated, “Learning to 
think critically is one of the most significant activities of adult life” (p. ix). Kummerow 
(cited in Naylor, 1984) suggested the most relevant tasks faced by adults include 
decision-making and problem solving. 

Gibbs (1985) reported that educators involved in the critical thinking movement 
would be able to direct their efforts more effectively if additional research were available 
to guide them. The review of literature for this study revealed reports of critical thinking 
done at the undergraduate level (e.g., Allen, 1995; Clarke, 1995; Gipe & Richards, 1992; 
Gokhale, 1995; Karabenick & Collins-Eaglin, 1996). None of these studies addressed 
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issues related to critical thinking in adult education. However, one study (Ruminski & 
Hanks, 1995) was of particular interest because of the instrument used. Although the 
population studied was journalism and mass communications faculty teaching at the 
undergraduate level, some items from the instrument were of value to investigate critical 
thinking issues with adult educators teaching graduate-level courses. 

This researcher investigated several components of critical thinking in the context 
of adult education. In order for adult educators to make more comprehensive decisions 
about critical thinking skills curriculum, instructional methods, and evaluation 
techniques, a working definition of critical thinking is needed. This study sought to 
contribute to the knowledge base on critical thinking for adult education by attempting to 
define critical thinking by identifying dimensions and characteristics of the construct. 
With an agreed-upon definition of critical thinking guiding their efforts, adult educators 
could better teach their students to think critically. 

Definition of Terms 

Terms used throughout this study are operationally defined as follows: 

Ability — what a person can or has the power to do. 

Adult — an individual performing social roles typically assigned by our culture to 
those it considers adults; that is, the roles of worker, spouse, parent, responsible citizen, 
soldier, and the like. A person is adult “to the extent the individual perceives herself or 
himself to be essentially responsible for her or his own life” (Knowles, 1980, p. 24). 
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Adult education — “the process whereby persons whose major social roles are 
characteristic of adult status undertake systematic and sustained learning activities for the 
purpose of bringing about changes in knowledge, attitudes, values, or skills” 

(Darkenwald & Merriam, 1982, p. 9). 

Adult educators -individuals who are currently teaching or previously have taught 
graduate-level courses in adult education in institutions granting graduate degrees in adult 
education. 

Construct — a postulated trait of human beings, such as critical thinking ability, 
intelligence, anxiety. Constructs are thought to underlie performance and explain its 
occurrence (Norris & Ennis, 1989). 

Creative thinking — “sensing difficulties, problems, gaps in information, missing 
elements, something askew; making guesses and formulating hypotheses about these 
deficiencies; evaluating and testing these guesses and hypotheses; possibly revising and 
retesting them; and finally communicating results” (Taylor cited in Garrison, 1991, p.47). 

Dispositions — habits of mind; habitual ways of behaving. 

Metacognition — awareness of one’s own cognitive processes; process of 
examining how oneself thinks and makes decisions. 

Limitations of the Study 

Data for this descriptive study were collected from adult educators who were 
currently teaching or have previously taught adult education classes in institutions in the 
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United States offering graduate-level degrees in adult education. The results of the study 
apply only to the population investigated and should not be generalized to other educators 
or trainers. 

Survey research methods were used to obtain adult educators’ perceptions of 
critical thinking skills and related topics. No commercial questionnaire was available; 
therefore, the researcher developed one. The questionnaire used for the study was not 
pilot tested although a draft of the instrument was reviewed by a panel of experts. 

Results of the questionnaire are limited by how respondents interpreted the items. 

The study investigated adult educators’ perceptions of three components of 
critical thinking: definitions of critical thinking, instructional methods used for teaching 
critical thinking, and methods of measuring students’ growth in critical thinking. 
However, these three are not the only components of critical thinking. The study is 
limited by the inclusion of these three components only. 

Summary 

Increasing attention is being given to critical thinking skills in response to the 
rapid rate of change in our complex world. In recent years, adult educators have begun to 
suggest that teaching critical thinking is an identifying characteristic and central function 
of adult education. A rationale for conducting this research was presented, and an 
overview of the study design was provided. Many issues introduced in this chapter will 
be addressed in more detail in the following chapters. 
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In Chapter 2, the related literature is reviewed concerning definitions, 
instructional methods, and methods of measuring gains in critical thinking skills. Chapter 
3 includes a detailed discussion of the research design and methodology utilized in the 
study. Chapter 4 contains a presentation and analysis of the data, and Chapter 5 discusses 
the results of the study and presents recommendations for future research. 
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Chapter 2 

REVIEW OF RELATED LITERATURE 

“The sign of a poor education ... is not ignorance. It is .. . the thoughtless habit 
of believing that one s unexamined, superficial or parochial opinions and feelings are the 
truth; or the habit of timid silence when one does not understand what someone else is 
talking about. ” 

(Grant Wiggins, 1989, p. 57) 

It is widely believed that the development of critical thinking skills is the most 
effective way to prepare individuals to live and work successfully in a complex world 
marked with rapid and constant change (Bangert-Drowns & Bankert, 1990; Brookfield, 
1987; Facione, 1990; Halpem, 1993; Howe & Warren, 1989; Kerka, 1992; Patrick, 1986; 
Terenzini, Springer, Pascarella, & Nora, 1995). Despite the fact that critical thinking 
skills are considered crucial to survival, research on critical thinking is limited (Terenzini, 
Theophilides, et al., 1984). McMurray et al. (1989) observed that despite recent progress 
in delineating a construct of critical thinking, the nature of critical thinking remains 
unclear. 

Current literature which investigates the definition of critical thinking, the 
efficacy of various instructional methods on the development of critical thinking, and the 
effectiveness of measurements of students’ growth in critical thinking skills is presented 
in this chapter. The first section reviews definitions of critical thinking; the second 
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section reviews instructional methods, and the third section reviews methods of 
evaluation. 



Definitions of Critical Thinking 

Operationalizing a concept such as critical thinking is not easy. In fact, one of the 
problems plaguing research in this area is the lack of a standard definition of critical 
thinking (Dixson, 1991; Garrison, 1991; Lipman, 1985; McMurray et al., 1989; 

Sternberg, 1985; Tucker, 1996). Yet, as McMurray et al.,(1989) and Facione (1990) 
pointed out, everything else rests on having a workable definition. “It is clear that efforts 
to teach critical thinking presume the ability to diagnose needs and to measure 
intervention effects, and measurement, in turn, presumes the ability to define the 
construct being measured” (McMurray et al., pp. 1-2). Facione stated that “a clear and 
accurate conceptualization of critical thinking is absolutely essential for the development 
of valid critical assessment tools and effective critical instructional programs” (p. 5). 

There are many variations on the definition of critical thinking. Bangert-Drowns 
and Bankert (1990) reported that critical thinking has been equated with a multiplicity of 
constructs, including intelligence, domain-specific expertise, problem solving, logic and 
sound reasoning, and other higher order mental activities. Some scholars, including 
Beyer (1985), have defined critical thinking in a narrow sense. Others (Ennis, 1987; 
Facione, 1990; Paul, 1982; Siegel, 1988) offered a broader definition, insisting that 
critical thinking consists of both skills and dispositions but debating exactly what those 
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skills and dispositions are. Some, including Ennis (1987), Garrison (1991), Lipman 
(1985), and Siegel (1988), proposed that good thinking involves creative thinking as well 
as critical thinking. Others maintained that creative thinking is separate and distinct from 
critical thinking. Overall, these disagreements reflect the complexity of critical thinking. 

The remainder of this section of Chapter 2 is organized around the following 
questions relative to a definition of critical thinking as described in the literature: 

1 . What is the goal of critical thinking? 

2. How should critical thinking be conceptualized? 

3. How do adult educators define critical thinking? 

4. What do the various definitions of critical thinking have in common? 

What Is the Goal of Critical Thinking ? 

As was discussed in Chapter 1 , the primary aim of critical thinking is to help 
individuals develop the skills and abilities that will enable them “ to live, work, and 
function effectively in our current and changing society” (Howe & Warren, 1989, p. 3). 
Examination of this aim yields some characteristics of critical thinking and thus helps 
define the construct. 

The goal of critical thinking requires learners to think for themselves which 
implies they develop the abilities to challenge assumptions, maintain a certain skepticism, 
and reserve judgment. Dewey (as cited in Meyers, 1986), one of the early advocates of 
critical thinking, associated critical thinking with “suspended judgment”, “healthy 
skepticism”, or “reflective thought” (p. 8). McPeck (1981) suggested that “perhaps the 
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most notable characteristic of critical thought is that it involves a certain skepticism, 
argument or suspension of assent toward a given statement, established norm or mode of 
doing things” (p. 6). 

While McPeck (1981) believed that skepticism is a key element of critical 
thinking, a study by Ruminski and Hanks (1995) resulted in a different perspective. Of 

172 members of the Association of Education in Journalism and Mass Communication 

I 

(AEJMC) responding to an open-ended question requesting definitions of critical 
thinking, less than one percent listed “skepticism”, “independence”, or “willingness to 
question”. 

In a manner similar to McPeck, Brookfield (1987) viewed critical thinking as 
more than indiscriminate questioning of ideas and activities. He stated that critical 
thinking involves a two-fold activity of “identifying and challenging assumptions”, and 
“exploring and imagining alternatives” (pp. 15, 229). Meyers (1986) called critical 
thinking “the ability to formulate generalizations, entertain new possibilities, and suspend 
judgment” (pp. 28-29). 

How Should Critical Thinking Be Conceptualized ? 

In attempting to define critical thinking, scholars have been concerned with the 
following issues: 

1 . Is critical thinking a product (i.e., a set of discrete skills) or a process? - 

2. Does critical thinking involve both skills and dispositions? If so, which 
skills and which dispositions? 
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3. Does critical thinking require collaboration with others? 

In response to the first question, Beyer (1985) defined critical thinking in a 
narrow sense, arguing that “critical thinking is not a process at least not in the sense that 
problem solving or decision making are processes; that is, critical thinking is not a 
unified operation consisting of a number of operations through which one proceeds in 
sequence” (p. 303). He defined critical thinking as a set of nine discrete skills (Beyer, 
1984), including: (1) distinguishing between verifiable facts and value claims; (2) 
determining the reliability of a source; (3) determining the factual accuracy of a 
statement; (4) distinguishing relevant from irrelevant information, claims, or reasons; (5) 
detecting bias; (6) identifying ambiguous or equivocal claims or arguments; (7) 
recognizing logical inconsistencies or fallacies in a line of reasoning; (8) distinguishing 
between warranted or unwarranted claims; and (9) determining the strength of an 
argument. Further, Beyer (1995) noted the word “critical” comes from the Greek word 
for criterion, kriterion, which he defined as a benchmark forjudging. He concluded that 
critical thinking is “judging the reasonableness or soundness and truthfulness of 
statements” (Beyer, 1995, p. 9), not taking things for granted. 

Others who defined critical thinking in a narrow sense, for example Dressel and 
Mayhew (1954), equated critical thinking with problem solving only. Kurfiss (1988) and 
D’Angelo (1971) also associated critical thinking with problem solving, but they pointed 
out differences between the two constructs. Kurfiss believed that critical thinking "is a 
form of problem solving, but a major difference between the two constructs is that critical 
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thinking involves reasoning about open-ended or ‘ill-structured’ problems, while problem 
solving is usually considered narrower in scope” (p. 45). Consistent with Kurfiss’ 
perspective, D’Angelo viewed problem solving as part of critical thinking but stated that 
critical thinking consists of more skills, e.g., intuition and creativity, than are used in the 
problem-solving approach. 

Despite the support for problem solving as a significant component of critical 
thinking, Ruminski and Hanks (1995) discovered that journalism and mass 
communication instructors at the undergraduate level did not define critical thinking that 
way. Responding to an open-ended question asking for definitions of critical thinking, 
only one person (out of 172 respondents) specifically named problem solving as central 
to the definition of critical thinking. 

A number of critical thinking experts disagreed that critical thinking is only a set 
of skills and maintained that critical thinking also involves dispositions. Paul (1990) 
argued that technical skills can be used simply to criticize the views of opponents and to 
reinforce one’s existing views. He characterized this view as “critical thinking in the 
weak sense” (p. 5). Critical thinking in the “strong sense” (Paul, p. 5) involved 
approaching issues from multiple perspectives and demanded open-mindedness to 
understanding points of view with which one disagrees. Another scholar who linked 
critical thinking with skills and dispositions was Siegel (1988) who described a “critical 
spirit” which includes the ability to reason but also “certain attitudes, dispositions, habits 
of mind, and character traits” (p. 39). 
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Among those who advocated for skills and dispositions was Ennis (1985) whose 
definition is the one most frequently quoted by scholars: “Critical thinking is reasonable, 
reflective thinking that is focused on deciding what to believe or do” (p. 45). Scholars 
may have confidence in this definition because they can interpret it broadly to include 
higher order thinking, problem solving, and metacognition (Schrag, 1992). Based on his 
broad definition of critical thinking, Ennis (1985) developed a taxonomy of critical 
thinking skills that includes thirteen dispositions and twelve abilities that together make 
up critical thinking. 

While many scholars defined critical thinking as involving both skills and 
dispositions, there is continuing debate about which skills and which dispositions 
constitute critical thinking. Ennis (1985) and Paul (1990) identified the dispositions they 
believed are most characteristic of critical thinking. In their collective judgment, critical 
thinkers habitually: are skeptical, fair-minded, open-minded; respect evidence and 
reasoning; value clarity and precision; consider different points of view, and willingly 
change a position when reason and evidence warrant. 

In 1990, the results of a two-year study of critical thinking by a body of 46 
scholars were published. Using a qualitative research methodology known as the Delphi 
method, the panel came to consensus regarding a definition of critical thinking which 
stated that good critical thinking includes a skill dimension and a dispositional dimension 
(Facione, 1990): 

We understand critical thinking to be purposeful, self-regulatory judgment which 

results in interpretation, analysis, evaluation, and inference. . . . The ideal critical 
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thinker is habitually inquisitive, well-informed, trustful of reason, open-minded, 
flexible, fair-minded in evaluation, honest in facing personal biases, prudent in 
making judgments, willing to reconsider, clear about issues, orderly in complex 
matters, diligent in seeking relevant information, reasonable in the selection of 
criteria, focused in inquiry, and persistent in seeking results which are as precise 
as the subject and the circumstances of inquiry permit. Thus, educating good 
critical thinkers . . . combines developing critical thinking skills with nurturing 
those dispositions which consistently yield useful insights and which are the basis 
of a rational and democratic society, (p. 3) 

Through the Delphi Project, the panel of critical thinking experts generated a list 
of six cognitive skills and sixteen sub-skills crucial to becoming a good thinker. Table 1 
contains a list of the skills and sub-skills outlined in the panel’s final report (Facione, 

' 1990, p. 12). 



Table 1 

Delphi Project’s List of Critical Thinking Skills and Sub-Skills 



Skill Sub-skill 

(1) interpretation categorization 

decoding significance 
clarifying meaning 

(2) analysis examining ideas 

identifying arguments 
analyzing arguments 

(3) evaluation assessing claims 

assessing arguments 



(table continues) 
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Skill 



Sub-skill 



(4) inference querying evidence 

conjecturing alternatives 
drawing conclusions 

(5) explanation stating results 

justifying procedures 
presenting arguments 

(6) self-regulation self-examination 

self-correction 



While Beyer’s (1985) listing of critical thinking skills (above) is not refuted by 
the Delphi Project, his list is subsumed under their notions of “identifying” and 
“analyzing” all aspects of arguments. The Delphi Project expanded Beyer’s list of skills 
to include the components of “interpretation”, “inference”, and “explanation” as well as 
the metacognitive skill of “self-regulation” by which one is asked to monitor one’s own 
possible bias toward an issue. 

Close examination of the skills identified by the Delphi Project demonstrates the 
46 experts’ belief that critical thinking skills are cognitive, not behavioral. Other critical 
thinking scholars agree with this position (e.g., Dixson, 1991; Halpem, 1993). 

The Delphi pafielists also came to consensus about the dispositions that good 
critical thinkers exhibit (Facione, 1990, p. 25) as shown in Table 2. 
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Table 2 

Delphi Project’s List of Affective Dispositions of Critical Thinking 

Approaches to life and living in general 

Inquisitiveness with regard to a wide range of issues. 

Concern to become and remain generally well-informed. 

Alertness to opportunities to use critical thinking. 

Trust in the processes of reasoned inquiry. 

Self-confidence in one’s own ability to reason. 

Open-mindedness regarding divergent world views. 

Flexibility in considering alternatives and opinions. 

Understanding of the opinions of other people. 

Fair-mindedness in appraising reasoning. 

Honesty in facing one’s own biases, prejudices, stereotypes, egocentric or sociocentric 
tendencies. 

Prudence in suspending, making or altering judgments. 

Willingness to reconsider and revise views where honest reflection suggests that change 
is warranted. 



Approaches to specific issues, questions or problems 

Clarity in stating the question or concern. 

Orderliness in working with complexity. 
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Approaches to specific issues, questions or problems 

Diligence in seeking relevant information. 

Reasonableness in selecting and applying criteria. 

Care in focusing attention on the concern at hand. 

Persistence though difficulties are encountered. 

Precision to the degree permitted by subject and circumstances. 



Having identified the cognitive skills and affective dispositions of critical 
thinking, the Delphi panelists cautioned that neither skills nor dispositions should be 
overemphasized at the expense of the other. Unless persons have the dispositions toward 
critical thinking, they are unlikely to apply their critical thinking skills appropriately in 
both their personal and their civic life (Facione, 1990). 

A comparison of the dispositions enumerated by the Delphi Project (Facione, 
1990) and those listed by Ennis (1985) and Paul (1990) shows considerable overlap. 

Another consideration in the definition of critical thinking is whether it requires 
collaboration with others. Several theorists have argued that critical thinking does not 
occur unless there is a sharing and interacting with others. For example, McPeck (1990) 
believed that critical thinking requires reflection, i.e., analyzing arguments, and then 
communication with others. He claimed this two-step approach helped avoid 
arbitrariness and also served as a procedure for testing judgments. Garrison (1992) 
summarized McPeck’ s position by stating the learner has a dual responsibility: (1) to 
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construct meaning (internal process), and (2) to justify that meaning through discourse 
with informed others (external process). Freire (1989) affirmed McPeck’s theory by 
claiming only dialogue with others is capable of generating critical thinking. 

Dewey (1933) also believed that knowledge results from the constant interplay 
between internal and external processes. He proposed five phases of reflective thought; 
the last phase was to test the hypothesis or understanding in the external world. 

Brookfield (1987) also suggested a five-phase model of critical thinking similar to 
Dewey’s. Brookfield’s phases included: a triggering event, appraisal of the situation, 
exploration to explain anomalies, development of alternative perspectives, and 
integration of perspectives into the fabric of living. According to Garrison (1992), 
Brookfield’s model begins and ends in the external world. In the middle are three 
reflective phases. During the integration phase, individuals act upon their perspectives by 
communicating with others. Thus, for Brookfield, meaning developed in isolation does 
not meet the criteria for critical thinking. It is only in the shared world that true meaning 
is achieved. 

Halpem (1984), on the other hand, viewed critical thinking as an internal process 
where one “takes new information, combines it with information stored in memory, and 
ends up with something more than and different from what one started with” (p. 4). 

In recent years, scholars have begun debating whether critical thinking includes 
creative thinking. Ennis (1985), Garrison (1991), Halpem, (1984), Lipman (1985), 
Nugent (1990), and Siegel (1988) argued that creative thinking is an integral part of 
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critical thinking. Viewing creative thinking as a process, Taylor (cited in Sternberg, 
1988) described creative thinking as “sensing difficulties, problems, gaps in information, 
missing elements, something askew; making guesses and formulating hypotheses about 
these deficiencies; evaluating and testing these guesses and hypotheses; possibly revising 
and retesting them; and finally communicating the results” (p. 47). Using this definition 
of creative thinking, it is easy to see similarities to previous descriptions of both critical 
thinking and problem solving. As Halpem (1984) stated: “Many decisions are involved 
in solving a problem, and generating satisfactory solution paths often requires 
considerable creativity” (p. 162). 

How Do Adult Educators Define Critical Thinking ? 

Garrison (1991) traced critical thinking as a central tenet of adult education to 
Eduard Lindeman, the father of adult education. Lindeman was a friend and colleague of 
John Dewey who developed an approach to critical thinking commonly referred to as the 
scientific method of inquiry. Darkenwald and Merriam (1982) described Dewey’s 
method as a problem-solving approach which involved several steps: clarification of a 
problem to be solved, development of ideas or hypotheses about the problem, and the 
testing of these hypotheses by an examination of evidence. 

Although Lindeman did not use the term, critical thinking (borrowed in part from 
Dewey) was a major aspect of his conceptualization of adult education. According to 
Garrison (1991), 
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Meaning for Lindeman, as with Dewey, was derived from the verification of ideas 
through the experience of living. As a result, the full cycle of critical or reflective 
thought espoused by Dewey is evident in Lindeman’ s work; a process beginning 
with identifying and challenging preconceptions or assumptions, exploring new 
ways of thinking, and then evaluating these ideas through the experience of living. 
(P- 296) 

Malcolm Knowles, Lindeman’ s chief disciple and advocate in contemporary adult 

education, adopted Lindeman’ s four basic assumptions about adult education. In their 

assumptions, both Lindeman and Knowles emphasized personal experience and problem 

solving which are closely associated with the critical thinking process. 

Although interest in critical thinking among adult educators may be traced to 

Lindeman and Knowles, Garrison (1991) believed the individual who explicitly brought 

critical theory to the attention of contemporary adult educators was Jack Mezirow. 

Mezirow (1981) authored an inductively derived mode of learning called “perspective 

transformation” (p. 6) which he described as 

the emancipatory process of becoming critically aware of how and why the 
structure of psycho-cultural assumptions has come to constrain the way we see 
ourselves and our relationships, reconstituting this structure to permit a more 
inclusive and discriminating integration of experience and acting upon these new 
understandings, (p. 6) 

Mezirow’ s (1981) cycle of perspective transformation is closely related to 
previously stated phases of critical thinking (e.g., appraisal of personal experience, 
integration of perspectives into everyday living). 

Brookfield, another prominent contemporary adult educator, is one of the leaders 
in the critical thinking movement in adult education. His definition of critical thinking as 
well as his five-phase model of critical thinking was described previously in this section. 
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Adult educators Darkenwald and Merriam (1982) suggested knowledge comes 
from critically reflecting and acting on life’s experiences. They believed a model of 
learning for adult education should include: recognizing a problem, analyzing it, 
discussing it in terms of other people’s experiences and available information, using 
information to formulate solutions, and acting upon the solution(s). Central to this model 
are critical reflection and problem solving which they perceive as the preferred methods 
of adult education. 

What Do the Various Definitions of Critical Thinking Have in Common ? 

It is interesting (though not surprising) that theorists, philosophers, psychologists, 
and other researchers and educators have been unable to come to consensus on a 
definition of critical thinking. An agreed-upon definition would be useful in helping 
identify critical thinking skills which in turn would be beneficial in identifying 
instructional methods and measurements for evaluating growth in critical thinking. 

While there is still no single agreed-upon definition of critical thinking, a 
synthesis of the literature indicated common ground among the various scholars of 
critical thinking, including: 

1 . Most scholars agreed that critical thinking includes skills or proficiencies 
on one hand and tendencies or dispositions on the other. 

2. Although opinions differed as to which skills and which dispositions 
constitute critical thinking, there was considerable overlap in the lists 
developed by various scholars.. (The reader is referred to Table 1 for 
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examples of critical thinking skills and to Table 2 for examples of critical 
thinking dispositions.) 

3. Many scholars believed critical thinking involves cognitive skills, not 
behavioral skills. 

4. Critical thinking involves both an internal process (reflection) and an 
external process (collaboration with others). 

5. Critical thinking seems to support a construct that involves two 
dimensions— analytic thought or the judging of information (that is, critical 
thinking) and the birthing of new ideas based on the combination of 
previous information (that is, creative thinking). 

6. Aspects of thinking that make it “critical” (Garside, 1996, p. 215) are: (a) 
thinking that is clear, precise, accurate, relevant, logical, and consistent; 

(b) thinking that reflects a controlled sense of skepticism or disbelief of 
any assertion, claim, or conclusion until sufficient evidence and reasoning 
are provided to support it conclusively; (c) thinking that takes stock of 
existing information and identifies holes and weaknesses, thereby 
certifying what is known and what is not known; and (d) thinking that is 
free from bias, prejudice and one-sidedness of thought. 

This section of Chapter 2 reviewed definitions of critical thinking. The next 
section summarizes the literature on the efficacy of various instructional methods on the 
development of critical thinking skills. 
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Teaching Critical Thinking 

Garrison (1991) suggested the most fundamental role of adult educators is to 
encourage and develop critical thinking not only because critical thinking is a central 
component of adult education but also because it is the one function that learners find 
most difficult to perform themselves. Given this challenge, one would expect adult 
educators to conduct research regarding how to teach critical thinking most effectively. 
However, few research studies at the adult education level have been done on this topic. 
As Smith (1980) discovered, educators have proposed a number of interesting ideas 
aimed at improving their students’ reasoning, but they have reported little experimental 
research on the effectiveness of their instructional strategies. McMillan (1987) observed 
there is little evidence that critical thinking skills are shaped by specific instructional 
variables. Reboy (1989) indicated that the problem may be that many of the critical skills 
currently found in various taxonomies are not “teacher friendly” (p. 41 1); that is, they do 
not lend themselves easily to instructional design and measurement. 

Although the purpose of the current research study was to examine adult 
educators’ perceptions of critical thinking, because of the limited research in adult 
education, the researcher expanded the literature review to include research studies that 
involved undergraduate students in a variety of disciplines. 

The remainder of this section of Chapter 2 is organized around the following 
questions relative to the teaching of critical thinking as described in the literature: 

1 . Can critical thinking be taught? 

erJc 




2. Should critical thinking be taught directly or indirectly? 

3. Which instructional methods are effective in teaching critical thinking 
skills? 

4. What conclusions can be drawn about the teaching of critical thinking 
skills? 



Can Critical Thinking Be Taught ? 

The claim that critical thinking skills can be taught is supported by a diverse body 
of evidence showing that “better thinking can be improved with appropriate instruction” 
(Halpem, 1993, p. 250). Chance (.1986) reviewed several thinking programs and 
concluded that good thinking is a skill that can be taught, and Kurfiss (1988) reported that 
critical thinking is a leamable skill. McPeck (1981) stated that to the extent critical 
thinking is a skill, it is teachable in much the same way other skills are teachable, namely 
through drills, exercises, or problem solving. 

Dixson (1991), on the other hand, argued that 

teaching is usually accomplished through example and explanation. Although 
some explanation is possible, it is difficult to “show” critical thinking. Since it is 
a cognitive, rather than a behavioral skill, we cannot directly observe the process. 
This makes it difficult to teach such a skill directly. It is far more likely that we 
can facilitate it. (p. 6) 

Dixson’ s viewpoint is more than a semantic difference. It reflects one of the 
strongly held tenets of adult education; namely, that educators should facilitate learning 
rather than impart knowledge (Brookfield, 1987; Candy, 1991; Darkenwald & Merriam, 
1982; Garrison, 1992; Knowles, 1980; Mezirow, 1981). According to adult education 
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theorists, the most important distinction between facilitating and teaching is that the 
facilitator engages the learner as an equal partner in every step of the learning process 
(Knowles, 1980). 

Using a simple vote counting procedure, the overall results from the current 
literature review were slightly positive for teaching critical thinking skills. Of the 26 
studies included in the review, 1 5 found significant differences or positive gains in 
critical thinking skills; seven found no significant differences, and the remainder yielded 
mixed results. 

Agreeing that critical thinking can be taught, scholars collectively have developed 

J 

a set of guidelines for teaching higher order or critical thinking skills as shown in Table 3 
(adapted from Halpem, 1984; Kerka, 1992; Kurfiss, 1988). 

Several of the guidelines in Table 3 reference characteristics of critical thinking as 
delineated in the definitions of critical thinking cited previously. Further, the guidelines 
confirm Kerka’ s (1992) observations that when educators teach critical thinking, other 
things change as well, particularly teaching methods, the role(s) of the teacher, and the 
classroom environment. For example, regarding changes in teaching methods, Kerka 
advocated building on what students already know which might be a way to “actively 
involve students in the learning” (Table 3, item 2). Regarding teachers’ roles, Kerka 
suggested teachers ask open-ended questions (Table 3, item 4). As far as changes in 
classroom environment, Kerka recommended opportunities for reflection on real life 
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Table 3 

Guidelines for Teaching Higher Order Thinking Skills 



Offer a rationale for learning the skills. 


7. 


Provide practice of thinking skills in 
multiple settings. 


Actively involve students in the 
learning. 


8. 


Use multiple learning strategies. 


Allow sufficient time for students to 
reflect on the questions asked or 
problems posed. 


9. 


Use examples that are similar to the 
situations in which the skills will be 
used. 


Ask open-ended questions. 


10. 


Teach for transfer. 


Promote interaction among students as 
they learn. 


11. 


Use intrinsic motivational techniques. 


Model problem-solving techniques. 


12. 


Promote metacognitive attention to 
thinking. 



situations and contexts. Table 3, item 9, advises teachers to “use examples that are 
similar to the situations in which the skills will be used”. 
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Should Critical Thinking Be Taught Directly or Indirectly ? 

A recurring topic in the literature is whether critical thinking should be taught 
directly or indirectly (Reboy, 1989). Browne, Haas, and Keeley (1978), de Bono (1983), 

Penner (1995), and Statkiewicz and Allen (1983) found that direct training combined 
with practice and reinforcement are needed to facilitate the development of critical 
thinking. For example, de Bono believed thinking skills can be taught directly, but in 
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order for these skills to be successfully transferred, they must relate to circumstances 
individuals face in their personal or professional lives, de Bono’s research stressed the 
element of practice to ensure students were comfortable with different strategies involved 
in critical thinking and problem solving. 

Results of a meta-analysis of 20 studies of the effects of explicit instruction for 
critical thinking by Bangert-Drowns and Bankert (1990) consistently favored programs 
that used explicit instruction methods. Further, intensive programs proved more effective 
than did programs providing only periodic training in critical thinking. Length of the 
treatment was unrelated to effectiveness. 

While results of the meta-analysis are useful regarding explicit versus indirect 
instruction, they do not help answer the related question about whether it is more 
effective to teach critical thinking skills in courses designed solely for that purpose or 
whether the instruction would be more effective if it were integrated into subject matter 
courses. 

Halpern (1993) claimed a broad-based, cross-disciplinary approach is most 

/ 

effective for critical thinking instruction. Further, she contended that critical thinking 
skills do not necessarily develop as a by-product of discipline-specific course work. The 
answer, she believed, is specially designed courses that focus on generic thinking skills 
using varied examples because such courses provide the ideal combination of skills 
training and practice with transferring skills. Chance (1986) recommended integrating 
critical thinking instruction into subject matter courses. 
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Before examining the results of the research studies included in this section of the 
literature review, a listing of the principles that guide critical thinking courses would be 
useful. Kurfiss (1988) summarized the common features of critical thinking courses 
taught across disciplines. These features apply as well to courses specifically designed to 
teach critical thinking skills (see Table 4). 

Table 4 

Common Features of Critical Thinking Courses 

1 . Critical thinking is a learnable skill; the instructor and peers are resources in 
developing critical thinking skills. 

2. Problems, questions, or issues are the point of entry into the subject and a source 
of motivation for sustained inquiry. 

3. Successful courses balance challenges to think critically with support tailored to 
students’ developmental needs. 

4. Courses are assignment centered rather than text and lecture centered. Goals, 
methods, and evaluation emphasize using content rather than simply acquiring it. 

5. Students are required to formulate and justify their ideas in writing or other 
appropriate modes. 

6. Students collaborate to learn and to stretch their thinking, for example, in pair 
problem solving and small group work. 

7. Several courses, particularly those that teach problem-solving skills, nurture 
students’ metacognitive abilities. 
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Four studies included in this literature review addressed the question of the 
efficacy of special courses designed specifically to teach critical thinking. Studies by 
Browne, Haas, et al. (1978), Davidson and Dunham (1996), and Logan (1976) supported 
the view that direct teaching of critical thinking skills through courses designed 
specifically to teach critical thinking significantly improves students’ thinking skills. On 
the other hand, a study by Ruminski and Hanks (1995) indicated that a large majority of 
journalism and mass communication faculty integrate instruction on how to think 
critically into the subject matter of their communication courses. 

Browne, Haas, et al. (1978) reported that after one academic quarter, college 
freshmen enrolled in a special course designed to teach critical thinking skills 
outperformed seniors (control group) on a standardized posttest. Davidson and Dunham 
(1996) studied the impact of a critical skills seminar on Japanese students enrolled in an 
English as a Second Language course. Students receiving the critical skills training 
scored significantly higher on the standardized posttest. 

Logan (1976) studied critical thinking skills as they were taught in an 
experimental course for college freshmen and sophomores. Results showed that students 
who took the experimental course were able to spot an average of 1 .79 fallacies among a 
possible ten on a scale measuring inclination to think scientifically. When specifically 
told to think scientifically, they spotted 2.35 fallacies. Graduate teaching assistants in the 
same department who had not taken the experimental course scored 1.11 and 1 .92 
respectively. 
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Which Instructional Methods Are Effective in Teaching Critical Thinking Skills ? 

McMillan (1987) pointed out that one of the primary means used to enhance 
critical thinking is classroom instruction. Young (cited in McMillan, 1987) added that it 
is assumed that if teachers use appropriate instructional methods and curriculum 
materials, students will improve their critical thinking skills. Yet educators continue to 
struggle to uncover instructional strategies that have a positive impact on students’ 
critical thinking. 

The literature included several studies investigating the efficacy of various 
instructional methods on undergraduate students’ thinking skills. Fourteen studies 
included in this literature review investigated one or more instructional methods for 
teaching critical thinking skills. Nine of these 14 studies resulted in statistically 
significant or positive results; four resulted in no statistically significant difference, and 
one produced mixed results. Instructional methods reviewed here that resulted in positive 
findings include: discussion classes, case study, structured inquiry, critical thinking 
environment, reflection (including “think aloud” activities), reasoning message strategies, 
collaborative learning, argumentation, annotation, logic, declarative sentence method, 
and out-of-class activities. Because the studies examined separate and distinct 
instructional methods, it is difficult to make meaningful comparisons and conclusions 
about particular methods. 

Having reviewed 27 studies that investigated the effects of instructional methods, 
courses, programs, and general college experiences on changes in students’ critical 
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thinking, McMillan (1987) concluded the results failed to support the use of specific 
instructional or course conditions to enhance critical thinking but did support the 
conclusion that college attendance improves critical thinking. McKeachie, Pintrich, Lin, 
and Smith (1986) disagreed with McMillan’s assessment. After completing an extensive 
review of the literature, they attributed improvement in critical thinking to three 
instructional variables: student discussion, explicit emphasis on problem solving, and 
explicit emphasis on methods to encourage development of metacognition. 

A previous review of the literature by McKeachie (1970) cited seven studies to 
demonstrate discussion classes are more effective than lecture classes in promoting 
retention and higher level thinking. McKeachie’ s review of studies also indicated that 
other variables, such as programmed learning, independent study, and simulation were 
found to be unrelated to critical thinking outcomes. Finally, McKeachie found student- 
centered classes rather than instructor-centered classes promoted higher level cognitive 
outcomes. Howe and Warren (1989) described student-centered classrooms as ones 
which involve students in paired problem solving, cooperative learning settings, 
simulations, debates, and critical reporting sessions. 

Allen (1995) researched the effects of case studies, integrative logs, and 
traditional lecture on undergraduate students’ achievement in an educational psychology 
survey course. Students in the case study group received instruction in a combined 
lecture-recitation format followed by in-depth discussion of a case. The discussion 
focused on developing problem-solving and decision-making skills. Students in the 



ERIC 



50 



41 



integrative log group were taught primarily through a traditional lecture-recitation format. 
Outside of classroom time, the students were required to produce log entries that 
attempted to develop reflective thought in a non-case manner. Students in the traditional 
lecture method were taught entirely through a lecture-recitation format without any 
specific reflective treatments. Results demonstrated that, when compared to more 
traditional instructional methods, the case study method significantly increased students’ 
content knowledge base. 

Baker and Anderson (1983) studied the effects of three inquiry methods on 
students’ critical thinking skills. Inquiry methods, according to Kurfiss (1988), are useful 
in teaching causal relationships and correcting misconceptions. Instructors using the 
inquiry method deliberately ask questions, select examples, and use entrapment strategies 
to elicit misconceptions in students’ thinking so they can be corrected. In the Baker and 
Anderson study, the structured inquiry method produced the highest percentage of gain, 
but the focused inquiry and open-ended inquiry also produced substantial percentage 
gains. These results support the findings of Kurfiss (1988) who examined numerous 
studies at the college level in which the inquiry method proved effective in improving 
critical thinking skills. 

Tien and Stacy (1996) examined three instructional environments: traditional, 
guided inquiry, and a course for non-science majors that emphasized critical reasoning. 
The traditional laboratory environment provided hands-on experiences in which the 
experiments were rote procedural exercises. Students were not required to engage in 
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meaningful problem-solving activities or examine evidence critically. For the guided 
inquiry environment, the researchers based their instructional methods on a modeling, 
coaching, scaffolding, and fading paradigm designed by Collins, Brown, and Newman. 
During the guided inquiry process, students were asked to predict outcomes, observe 
data, and explain results. The third environment in the study was a critical reasoning 
course designed for non-science majors. Students were asked to apply chemistry to 
everyday problems and evaluate chemistry-related studies published in newspapers and 
magazines. The results of the Tien and Stacy study showed the critical reasoning 
environment was more successful in fostering inquiry skills than the other two 
environments. Critical reasoning students outperformed both guided inquiry and 
traditional students in regard to explaining scientific procedures and offering relevant 
improvements for fabricated studies. 

Through a multi-tiered study, Koehler and Neer (1996) researched the 
relationship and impact of trait argumentativeness (“one’s disposition to either approach 
or avoid an argument”, p. 6), argumentative flexibility (“one’s disposition to argue in 

either a flexible or collaborative manner, or an inflexible manner”, p. 6), and self- 

) 

reported perception of critical thinking dispositions on critical thinking competence. Two 
separate surveys, including standardized tests for argumentation and critical thinking 
skills, were administered five weeks apart. Results showed that neither argumentative 
style nor perceived critical thinking competence (as measured by self-report) impacted 
positively on critical thinking competence scores. 
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Walters and Strode (1991) investigated the impact of training in annotation 
writing on students’ comprehension and summary writing. The researchers described 
annotation writing as “summarizing written information as well as commenting or 
reacting to a piece of writing” (p. 2). Students in the experimental group received 
annotation training which allowed for practice, revision, and sharing. Using a 
nonequivalent control group design, the researchers reported no significant difference in 
critical and evaluative thinking skills between students receiving the special training and 
students receiving a more traditional type of instruction. 

Darkenwald and Merriam (1982) described critical reflection as one of the 
preferred methods of adult education and emphasized that opportunities for reflection 
should include discussion, questions, and collaborative exploration of differing 
viewpoints. The use of reflection as a method of enhancing critical thinking skills was 
investigated in ethnographic studies by Clarke (1995) and Gipe and Richards (1992). 
Clarke’s study followed four science student teachers during a 13 -week practicum. The 
student teachers were videotaped during “reflective teaching cycles” (p. 246), and the 
transcribed tapes were given to students for their analysis regarding themes and incidents 
of reflection. The researcher was interested in learning: (1) What do student teachers 
reflect upon? (2) What precipitates the student teachers’ reflections? (3) What factors 
enhanced or constrained the student teachers’ reflections? Results included the 
documentation of 1 5 reflective themes across the four cases. In all, 47 factors were 
documented that either enhanced or constrained student teacher reflection. 
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In another ethnographic study, Gipe and Richards (1992) also studied prospective 
teachers and reflective thinking. In this study, 23 elementary education majors engaged 
in an early field experience and wrote their thoughts and feelings about teaching for 15 
weeks. The dialogue journals were evaluated by university supervisors who looked for 
the number of reflective statements per journal entry. These results were compared to the 
evaluations of teaching ability made by supervising teachers and the university’s student 
teacher supervisor. The findings supported the assumption that reflective thinking helps 
improve teaching ability. 

Lavoie (1991) investigated the effect of “if-then” logic in making successful 
predictions. Participants were pre-service college students enrolled in methods classes. 
The experimental group used “think-aloud” interviews to solve problems; the control 
group solved predication-problem sheets silently in class. The think-aloud interviews 
were more effective than traditional methods for identifying the knowledge of individual 
students. 

Three studies included in this literature review studied collaborative learning at 
the undergraduate level and obtained mixed results. Garside (1996) found no significant 
differences when he compared learning resulting from traditional teaching methods with 
learning resulting from collaborative groups. Gokhale (1995) and Karabenick and 
Collins-Eaglin (1996), on the other hand, found that collaborative learning is beneficial to 
student learning. 
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The Garside (1996) study involved 118 students enrolled in an introductory 
interpersonal communications course at a medium-sized university. Students in the 
experimental group (three classes) were organized into small groups and were given a list 
of questions that required them to talk about the concepts the instructor wanted to cover 
in class. The control group received the concepts in a lecture format. No significant 
differences were found between the two instructional teaching strategies. However, the 
researcher offered several reasons why this result may have occurred: (1) the lecture is 
still the dominant method of instruction in college classrooms; therefore, students have 
less experience with group discussion; (2) in this study, the group’s discussion process 
was not structured, and the students may have lacked experience with group discussion in 
general; and (3) the study is based on a limited sample size in one course. 

In another study of collaborative learning, Gokhale (1995) examined the 
difference in achievement by students engaged in drill and practice and students involved 
in collaborative leaming/discussion groups. Findings demonstrated a significant, 
difference in achievement when discussion group students were tested on critical thinking 
items, but no significant difference was found regarding drill and practice. The 
researcher concluded that collaborative learning fosters the development of critical 
thinking. 

Several scholars (Beyer, 1985; Dixson, 1991; Halpem, 1984; McPeck, 1990) 
would applaud Gokhale’s findings. While Dixson did not conduct research on the 
subject, she reviewed the effects of group discussion and believed this method is highly 
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conducive to the development of students’ critical thinking skills. McPeck suggested that 
teachers need to change their methods of presentation from a didactic mode to a 
discussion mode of teaching and assessment in order to emphasize critical thinking skills. 
Beyer (cited in Garside, 1 996) described the characteristics of classrooms that reinforce 
and support critical thinking: 

Students feel free to risk, challenge, and question; there is student-to-student 
interaction focused on information processing, where students consider the ideas, 
contributions, and arguments of peers; teachers don’t “tell”, rather, they help 
students critically analyze ideas; students are encouraged to become active 
learners rather than passive recipients of information; and students take 
responsibility for their own thinking and learning, (p. 216) 

Karabenick and Collins-Eaglin (1996) studied two features that impact learning: 

learning goals and incentive structures. The researchers were particularly interested in 

the association between goals and incentives and students’ use of learning strategies 

known to facilitate student performance. Participants were asked to complete a 

questionnaire described to them as a way to acquire more information about their 

learning environment. Based on responses from 1,037 participants, the researchers found 

the more that students indicated that grades in their classes were based on group 
performance (rather than individual performance) the more they reported using 
critical thinking. In addition, perceived emphasis on grades was inversely related 
to students’ use of cognitive strategies of elaboration and critical thinking. Most 
notable, however, is that more emphasis on student cooperation to learn the 
material was related to greater student use of learning strategies of elaboration, 
metacognition, and critical thinking. Critical thinking was also more evident in 
classes in which grades were based on group performance. There was no 
evidence, however, that emphasis on competition or individual accomplishment, 
or learning and comparative ability emphasis was related to a class’ use of 
learning strategies, (p. 5) 
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Smith (1977, 1983), like Karabemick and Collins-Eaglin (1996), investigated 
learning atmospheres. Specifically, he was interested in student participation in class 
discussion, teacher encouragement, and peer-to-peer interaction as they impact student 
grades. In his 1977 study, Smith found statistically significant associations between high 
scores on the Watson-Glaser Critical Thinking Appraisal and greater student participation 
in class discussion, higher encouragement by the teacher, and higher peer-to-peer 
interaction. In a later study (1983), Smith, after one semester, found no significant 
relationship between students’ critical thinking skills and the three factors being studied. 

In a mass communications course, Meiss and Bates (1984) used three distinct 
instructional methods to assess the development of critical thinking and affective 
behavior. The independent variables were three reasoning guides: declarative sentences, 
questions, and topical outlines. Students instructed according to the first two guides were 
introduced to a hierarchical thinking process which included application, analysis, 
synthesis, and evaluation of basic mass communication concepts and related those 
cognitive processes to problem solving. Students taught by the topical outline method 
served as the control group. The researchers found a statistically significant 
improvement in critical thinking by students exposed to the declarative sentence method 
but no significant difference for students exposed to the question guide method or the 
topical outline. The researchers suggested that strategies based on the discovery theory 
of learning, such as the declarative sentence method which encourages student 
participation, self-awareness and informed guessing, foster a growth in cognition. 
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Statkiewicz and Allen (1983) introduced out-of-class practice exercises to 
examine development of students’ analytical skills and transferability of skills to new 
situations. Their findings indicated that with practice analytical skills improve and these 
skills will transfer to new and unfamiliar problems. However, Gibbs (1985) in his review 
of this study cautioned that the authors’ inferences were weakly supported due to a lack 
of a control group or even a comparison group. 

Although the research studies cited above were conducted at the undergraduate 
level and in various disciplines, they can help inform adult educators about the teaching 
of critical thinking. What is known about teaching critical thinking in adult education 
comes primarily from theorists. Adult educators, in general, agree with Knowles (1980) 
that adults’ experiences are a rich resource for learning. Consequently, Feuer and Geber 
(1988) among others advocated for participatory, learner-centered techniques that draw 
on learners’ experiences. Lanese (1983) encouraged adult educators to use the 
experiences of adults as the primary curriculum in adult education classes. 

Knowles (cited in Mezirow & Associates, 1990) recommended tapping into the 
experiences of learners by using experiential techniques such as group discussion, 
simulation exercises, problem-solving activities, case method, and laboratory methods. 
Mezirow and Associates noted that facilitators are effective in using learners’ experiences 
when they 

encourage learners to look at problems from many perspectives, challenge one 
another and themselves, probe connections, try out new behaviors, see and 
confront their own dysfunctional behavior, act when they would rather talk, and 
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reflect some more when they are ready to act prematurely to solve a problem that 
has not been thoroughly considered, (p. 44) 

While adult educators are in agreement that experiential techniques are effective 
with adult learners, they also recognize that no one instructional strategy or method can 
be effective for all learners or all learning goals. They insist facilitators of adult 
education must have a broad repertoire of instructional methods and be able to choose the 
most appropriate method determined in part by the purpose, objectives, and content of the 
lesson as well as the learning styles of the adult learners (Hayes, 1993). Merriam (1984) 
proposed contract learning, experiential learning, portfolios, and self-pacing as among the 
most effective instructional techniques for use with adult learners. 

Although adult educators agree that teaching adults should be approached in a 
different way from teaching pre-adults, there have been few efforts to test whether 
educators actually use a different style when teaching adults (Imel, 1989). Two studies 
(Beder & Darkenwald, 1982; Gorham, 1984, 1985) examined this area by investigating 
the following questions: Do teachers teach adults in a different way? If so, what are 
these differences? In both studies, subjects were educators who taught both adults and 
pre-adults. In the Beder and Darkenwald study, information was collected solely through 
a self-report questionnaire. Gorham used an adaptation of Beder and Darkenwald’ s 
instrument for the initial phase of her study, then followed up with classroom 
observations of a small number of her sample for a second phase. 

Both studies investigated educators’ perceptions of the learning differences 
between adults and pre-adults and found that educators believed adults to be significantly 
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more intellectually curious, motivated to learn, willing to take responsibility for their 
learning, willing to work hard at learning, clear about what they want to leam, and 
concerned with the practical applications and implications of learning than were children 
and adolescents. In both studies, respondents reported significant differences in teaching 
styles. As compared to teaching children and adolescents, when teaching adults, 
respondents reported they spend less time on discipline and giving directions, provide 
less emotional support to students, structure instructional activities less tightly, and vary 
their teaching techniques more. Beder and Darkenwald (1982) also found significant 
differences in adult classes in greater use of group discussion, more adjustment in 
instructional content in response to student feedback, and a greater relationship of class 
material to student life experiences. 

Gorham’s (1985) follow-up classroom observations did not verify the self- 
reported information in the Beder and Darkenwald (1982) and Gorham (1984) 
questionnaires. The follow-up observations yielded several findings, including: (1) the 
use of directive teacher behavior was essentially the same with adults as with pre-adults, 
and (2) adults and teachers with more formal training in adult education tended to use 
student-centered approaches the least. However, educators did change the classroom 
environment to a more nontraditional, less formal room arrangement when they were 
teaching adults. 

Educators involved in critical thinking instruction have studied the impact of a 
college education on students’ critical thinking abilities. Eight studies in this literature 
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review (Browne & Keeley, 1988; Givens, 1976; Keeley, Browne, & Kreutzer, 1982; 
Lehman & Dressel, 1963; Lehman & Nisbett, 1990; Pascarella, Bohr, Nora, & Terenzini, 
1996; Ruminski & Hanks, 1995; Terenzini, Springer, et al., 1995) investigated the impact 
of college attendance on critical thinking skills. 

A descriptive study by Ruminski and Hanks (1995) indicated that the majority of 
journalism and mass communication instructors responding to a questionnaire either 
strongly agreed (13%) or agreed (55%) with the statement: “A four-year college 
education improves most college students’ critical thinking skills” (p. 8). Results of other 
studies conflicted with this finding. 

The studies yielding mixed results or no significant results will be presented first. 
Browne and Keeley (1988) recruited 37 college seniors to respond to an essay question 
designed to measure the cognitive processes individuals go through as they engage in 
critical evaluation. The results showed many seniors lack fundamental critical thinking 
skills. Keeley et al. (1982) obtained results that favored seniors over freshmen in a test of 
critical thinking skills, but the researchers expressed concern that 40% to 60% of seniors 
failed to provide a single example of a logical flaw, significant ambiguity, or misuse of 
data in a written passage. 

Lehman and Nisbett (1990) conducted a longitudinal study to determine if an 
undergraduate education improves reasoning about everyday-life problems and whether 
some undergraduate training enhances reasoning more than others. Participants were 
tested in the first term of their freshman year and in the second term of their fourth year 
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of college. Though no statistical differences were found pre- to posttest, the results were 
promising, and the researchers contended there is optimism for believing that reasoning 
can be taught and that different undergraduate disciplines can teach different kinds of 
reasoning and in differing degrees. 

Givens (1976) tested 40 faculty members and their students in four universities to 
answer questions about the cognitive level of classroom discourse, the difference in 
critical thinking skills between professors and students (basic and advanced), and the 
difference in critical thinking skills between students in smaller and larger classes. The 
findings showed student and faculty discourse averaged on the lowest cognitive level. 
There was no statistically significant difference between critical thinking scores of basic 
and advanced students. Scores were higher for students in small classes and for students 
in larger institutions. There was a positive association between “analysis” scores by 
professors and performance by students but no consistent relationship between the 
cognitive level of professors and corresponding cognitive level of students in their 
classes. 

The four studies described above cause concern about the effectiveness of a 
college education in improving students’ thinking skills. On the other hand, three studies 
provided evidence that a college education, in and of itself, can improve students’ 
thinking skills. In a longitudinal study of Michigan State University students, Lehman 
and Dressel (1963) found a statistically significant improvement in critical thinking on 
freshman-to-sophomore, sophomore-to-junior, and junior-to-senior comparisons. 
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Pascarella et al. (1996), conducting a longitudinal study of 2,076 first-year full- 
and part-time students at two- and four-year colleges, also found that college attendance 
has a positive impact on the development of critical thinking. In addition, the more a 
student was exposed to the academic experience of college, the larger the net positive 
impact on his/her growth in critical thinking. 

Terenzini, Springer, et al. (1995) studied the extent to which critical thinking is 
shaped by students’ formal academic activities and out-of-class experiences. The results 
demonstrated that both classroom/instructional activities and out-of-class experiences 
make positive, statistically significant contributions to gains in critical thinking. 

Though not directly related to the discussion of impact of college attendance on 
critical thinking skills, Garcia and Pintrich (1992) conducted a research study which has 

implications for adult educators. They investigated these research questions: What are 

* 

the relationships among motivation, learning strategies, and critical thinking? What is the 
relationship between classroom experience and critical thinking? Study results provided 
support for positive relationships among motivation, cognitive engagement, and critical 
thinking. Collaborative learning and challenging course work were also positively 
related to critical thinking. 

What Conclusions Can Be Drawn about the Teaching of Critical Thinking Skills ? 

Given the diversity of research questions, research designs, and instructional 

/ 

methods used in the research studies included in this literature review, there are few 
conclusions that can be drawn about the efficacy of teaching critical thinking skills at the 
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college level. Further, as Gibbs (1985) pointed out, much of the extant research utilized 
weak designs, particularly lack of randomization, and therefore results must be 
interpreted with caution. 

However, based on the studies included in this literature review, the following 
general observations can be made about teaching critical thinking: 

1 . Critical thinking can be taught with appropriate instruction. 

2. There is a growing body of literature that supports the direct teaching (as 
opposed to the indirect teaching) of critical thinking skills. 

3. There is little evidence to support or refute the efficacy of selected 
instructional methods. 

4. Researchers have focused on other related issues which continue to cause 
debate: Where in the curriculum should critical thinking skills be taught? 
Are critical thinking skills transferable to problems that occur outside the 
classroom? 

5. Critical thinking skills are highly valued, but there is an insufficient body 
of knowledge to inform educators about what works and what does not 
work in the teaching of critical thinking skills in higher education and 
adult education. 

This section of Chapter 2 reviewed the literature related to teaching critical 
thinking skills. The next section will review the literature related to evaluating growth in 
critical thinking. 
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Evaluating Critical Thinking 

There is a close relationship among the three critical thinking components 

examined in this study: definitions of critical thinking, strategies for teaching critical 

thinking, and methods of measuring gains in critical thinking. Facione (1990), in writing 

the final report of the Delphi Project, stated that “a clear conceptualization of critical 

thinking is absolutely essential for the development of valid critical thinking assessment 

tools and effective critical thinking instructional programs” (p. 5). Tucker (1996) 

affirmed Facione’s conclusion and emphasized that 

Many of the unresolved issues in creating a broadly useful model for assessing 
and teaching critical thinking are unlikely to be resolved until there is 
considerably more empirical evidence about what critical thinking means . . . and 
what components of this meaning are responsive to educational intervention, (p. 

7) 

Lacking an operational definition of critical thinking, it is not surprising that few 
valid and reliable assessment strategies and methods have been developed. After 
searching the literature for research studies examining critical thinking outcomes for 
college students and finding only 27, McMillan (1987) concluded the paucity of research 
suggested greater attention needs to be directed to the measurement of critical thinking 
skills. 

Educators and evaluators agreed that questions surrounding the issue of 
evaluation of critical thinking are diverse and difficult (Ennis, 1993; Facione, 1990; 
Halpem, 1993; Norris, 1988; Tucker, 1996). Furthermore, they recognized that currently 
available measures have many limitations. Therefore, these scholars have recommended 
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that better critical thinking measures be constructed and considerable empirical research 
be conducted in the field (Norris, 1988). 

The remainder of this section of Chapter 2 is organized around the following 
questions relative to the measurement of critical thinking skills as described in the 
literature: 

1 . What are the issues surrounding the evaluation of critical thinking skills? 

2. Which methods are effective in measuring gains in critical thinking? 

3. What conclusions can be drawn about the evaluation of critical thinking 
skills? 

What Are the Issues Surrounding the Evaluation of Critical Thinking Skills ? 

Educators committed to teaching critical thinking are interested in knowing what 
to measure, how to measure, when to measure, and how often to measure changes in 
critical thinking skills. They are also concerned about whether critical thinking skills are 
transferable and whether gains in critical thinking last. The literature cited experts’ 
opinions as well as limited empirical evidence on these issues. 

As was discussed earlier in this chapter, critical thinking experts have offered 
various definitions of critical thinking. For example, some consider critical thinking to 
be a process, while others defined it as a product. How critical thinking is defined 
determines what will be measured through critical thinking assessments. Thus, prior to 
designing assessment tools, test makers must decide if critical thinking is a process or 
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product, a skill or disposition, a cognitive or behavioral skill, an internal or external 
process. 

Measuring growth in critical thinking skills is challenging, particularly for those 
who define critical thinking as a process. Scholars such as Facione (1990), Kerka (1992), 
and Norris (1988) were more concerned with whether sound thinking processes were 
displayed than with correct answers to test questions. Kerka reported that “existing right 
and wrong answer approaches to testing are clearly inadequate” (p. 3). Facione observed 
that sound critical thinking assessment should target the quality of critical thinking 
students put into arriving at their answers. Norris argued that “Different examinees can 
v arrive at different answers using equally justifiable processes of inference making” (p. 

132). 

Contrary to current practice, the 46 critical thinking experts participating in the 
Delphi Project recommended frequent measurements of gains in critical thinking and 
utilization of multiple assessment methods in order to compile a composite picture of the 
subject and to cross check the result(s) of any one assessment technique (Facione, 1990). 

Another concern is the timing of measurements of critical thinking. Flalpem 
(1993) pointed out that cognitive growth is a gradual and cumulative process, making 
assessment more difficult. Flalpem (1993) and McMillan (1987) noted, and the research 
of Terenzini, Springer, et al. (1995) confirmed, that it is unrealistic to expect a huge gain 
in thinking abilities that can be attributable to one course that is a quarter or semester in 
length. Brookfield (1990a) concurred, stating a great deal of what is learned occurs long 
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after the course is finished. Yet many empirical studies are based on outcomes after one 
course on critical thinking. 

Halpem (1993) was also concerned with whether or not gains in critical thinking 
last over time. Specifically, she posed the following research question: “Do students 
who show improvement in their ability to think critically maintain these gains over time, 
or do they revert to easier and less effective methods of thinking” (Halpem, p. 243)? 
Halpern believed an ideal measure must extend beyond the semester or quarter of 
instruction so the evaluator can determine if any lasting gains are obtained. Yet, 
according to current practice, subjects are not re-evaluated at intervals following the 
completion of instruction on critical thinking. 

Additionally, there is concern about whether gains in critical thinking in one area 
are transferable to another area. Kerka (1992) wrote that cognitive research demonstrates 
that learning is not automatically transferred to new settings. Research by McMurray et 
al. (1989) suggested that critical thinking skills may not generalize across content 
domains. McMillan’s (1987) review of several empirical studies also indicated critical 
thinking skills may not transfer. Consequently, he recommended measures of critical 
thinking skills that are curriculum specific. Ennis (1993) examined critical thinking 
instruments and concluded there were no subject-specific critical thinking tests even 
though the National Academy of Education recommended there be a strong effort to 
develop subject-specific higher order thinking tests. 
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Which Methods Are Effective in Measuring Gains in Critical Thinking ? 

Facione (1990) and Norris (1988) stated that no technique for gathering 
information about thinking is infallible, and each has limitations which must be explored 
and understood. It is beyond the scope of this thesis to discuss the limitations of the 
existing critical thinking instruments. However, a brief discussion of some of the 
criticisms of the most frequently used measurements is in order. 

McMillan (1987) reviewed several research studies and concluded that currently 
available instruments rely on a single measure of critical thinking and may not be 
sufficiently discriminating to measure changes in critical thinking. He claimed “what is 
needed is a set of multiple measures of critical thinking that can be used to triangulate the 
results” (p. 15). He suggested measures such as student and teacher perceptions, essay 
answers, and locally devised instruments could be used in addition to appropriate 
standardized tests. 

While there are many critical thinking assessments available, most of them are 
multiple choice tests (Ennis, 1993). Ennis (1993) and Halpern (1993) expressed concern 
that multiple choice critical thinking tests are not sensitive enough to pick up on subtle 
changes in critical thinking abilities and often do not attempt to measure dispositions to 
think critically. For example, many instruments do not test for dispositions such as open- 
mindedness and willingness to change a position when reason and evidence warrant. 

Norris (1988) expressed concern that standardized tests may penalize students 
who are thinking critically because: (1) the students may make assumptions different 
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from the assumptions made by the test designers, or (2) the students’ background 
knowledge differs from that of the test makers. Further, Norris was apprehensive about 
objective critical thinking tests because they measure only the products of thinking. He 
believed that for really satisfactory evaluation of critical thinking ability, one must 
employ such techniques as essay testing, naturalistic observation, and one-on-one oral 
testing. 

The Watson-Glaser Critical Thinking Appraisal (WGCTA) is the most commonly 
employed objective measure of critical thinking (McMillan, 1987). However, there is a 
growing body of literature which cites problems with the instrument. For example, 
Dowling (1990) suggested the five sub-skills reportedly measured by the WGCTA are 
not grounded in any particular theory of human cognition. Browne, Haas, et al. (1978) 
suggested the WGCTA, which is a multiple choice test, may measure the ability to 
recognize a valid syllogism but may not test the ability of students to apply valid 
deductive reasoning to a problem. 

McMillan (1987) found the WGCTA technically weak and described problems 
with construct validity, normalization data, and the assessment of inferences as true or 
false rather than valid or invalid. Based on his review of 27 studies, 1 6 of which used the 
WGCTA, McMillan concluded that when the WGCTA is used as the dependent measure, 
it is likely nonsignificant measures will be found. 

In place of the WGCTA, Dowling (1990) recommended using the “reflective 
judgment” approach which he claimed corrects for the weaknesses of the WGTCA 
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because it: (1) has a clear foundation in cognitive developmental theory; (2) has been 
validated by a growing body of empirical data; (3) measures skills that are teachable; and 
(4) deals with problem-solving skills that are most useful to the real world. 

Browne, Haas, et al. (1978) were dissatisfied with the WGCTA because of the 
lack of correlation between student performance on the test and evaluation of classroom 
experience and paper assignments. They devised an essay exam with open-ended 
questions and developed explicit evaluation criteria for which they established reliability. 
The test measures not only whether students recognize critical thinking but whether they 
can engage in it. 

To investigate the methods used by journalism and mass communications 
instructors teaching at the undergraduate level, Ruminski and Hanks (1995) asked 
participants to note which of two ways they evaluated students’ critical thinking. Nine 
percent of the respondents reported they used a standardized test, but none of them 
named a specific test. The remaining respondents (91%) indicated they measured critical 
thinking using writing assignments. 

Other concerns about critical thinking measurements include validity and 
reliability (Facione, 1990; Halpem, 1993). Facione provided definitions of validity and 
reliability as they relate to assessment of critical thinking and called for the development 
of valid and reliable assessment strategies from which educators can draw reasonable 
inferences about students’ critical thinking, in contrast to their domain-specific 
knowledge or other academic abilities (such as reading or writing). 
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Halpem (1993) reported that 

In general, multiple choice tests tend to have problems with validity, and open- 
ended verbal and written examinations tend to have problems with reliability. 

The current trend toward open-ended assessments such as portfolios, interviews, 
focus groups, and responses to hypothetical situations all suffer from the problem 
of unknown reliability and validity, (p. 244) 

While many scholars described limitations of existing measurements, they also 
identified emerging assessment methods or strategies which may be more effective in 
measuring gains in critical thinking. Kerka (1992) expressed hope that new forms of 
evaluation such as the tailored response test, stimulated recall, scenario analysis, and 
concept mapping will be more effective in measuring increases in critical thinking. She 
also indicated that existing methods such as true-false, multiple choice, and essay can be 
adapted by having students indicate why an answer is false, how two things are similar or 
different, or by requiring evaluation or critique. 

Facione (1990) believed there are several ways, in theory at least, to measure 
proficiency in critical thinking, including observation, comparison of the results of 
carrying out a given skill against some predetermined criteria, and interviews with 
individuals to obtain their descriptions of the procedures and judgments they used as they 
exercised a particular thinking skill. Halpem (1993) suggested that good assessment of 
critical thinking should be based on “simulation scenarios” (p. 242) that are similar to 
situations the students will encounter outside the classroom. 

Despite the difficulties of measuring gains in critical thinking, Halpem (1993) 
concluded there are at least seven different forms of outcome evaluations that generally 
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support the idea that thinking skills can be improved. These include formal evaluations, 
student self-reports, gains in IQ scores, cognitive growth and development, expert-like 
mental representations, a cognitive skills approach, and measures of spontaneous transfer. 

Smith (1977) believed that limitations in evaluating gains in critical thinking were 
responsible for the failure to find meaningful differences among teaching methods. 
Consequently, he designed a different approach to measure gains in critical thinking. In 
addition to using the WGCTA as pre- and posttest, he used the Chickering Critical 
Thinking Behaviors Inventory to assess the impact of different classroom processes on 
critical thinking. The Chickering self-report instrument asked students to report the 
percentage of time spent on each of six activities (defined according to Bloom’s 
taxonomy) while studying for the particular course in question. The results were not only 
statistically significant but dramatic, leading Smith to conclude that “improved research 
strategies do indeed add to our knowledge of instruction” (p. 187). 

What Conclusions Can Be Drawn about the Evaluation of Critical Thinking Skills ? 

Critical thinking is a complex construct, and a variety of ways of measuring it 
have been advanced (McMillan, 1987). Educators interested in developing and 
measuring students’ critical thinking skills have been impeded by two factors: (1) there 
is a paucity of research to guide their efforts; and (2) available critical thinking measures 
have significant limitations. 

A synthesis of the literature suggested the following general findings about the 
evaluation of critical thinking skills: 
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1 . Available measurements of critical thinking have serious weaknesses. 

They tend to focus on products (i.e., correct answers), not process (i.e., the 
ability to think critically). They tend to rely on a single measure of critical 
thinking. Most are multiple choice tests and incapable of measuring 
significant components of critical thinking such as dispositions to think 
critically. Commercially available tests often do not consider potential 
differences in background knowledge and assumptions between test takers 
and test makers. 

2. Contrary to current practice, assessment of critical thinking should be 
done frequently and over time, rather than expecting significant results in 
a short period of time. 

3. Good assessment should use multiple measures rather than relying on a 
single measure. Suggested alternative measures include new forms of 
evaluation being developed such as the tailored response test, stimulated 
recall, scenario analysis, and concept mapping as well as familiar forms 
such as portfolios, interviews, focus groups, observations, and case 
studies. 

4. Care should be given that assessments are valid, reliable, and fair. 

5. Because critical thinking skills may not generalize across content domains, 
subject-specific critical thinking tests are needed. 
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Summary 



This chapter has reviewed pertinent literature in the area of critical thinking as it 
relates to definitions, instructional methods, and evaluation techniques. Although no 
consensus has been reached regarding a standard definition of critical thinking, scholars 
generally agree that critical thinking includes skills and dispositions, cognitive skills 
rather than behavioral skills, internal and external processes, and creative thought. There 
is also general agreement about which aspects of thinking make it “critical”. Garside 
(1996) has summarized these aspects which include: (1) thinking that is clear and 
logical; (2) thinking which reflects skepticism until sufficient evidence is provided; and 
(3) thinking that is free from bias. 

While critical thinking skills are highly valued, there is little empirical research 
that informs educators about what works and what does not work in the teaching of 
critical thinking skills. Further, much of the available research used weak designs, 
particularly lack of randomization, and therefore results must be interpreted with caution. 
Flowever, based on a review of the literature, some general observations can be made 
about teaching critical thinking. First, critical thinking can be taught (or facilitated) with 
appropriate instruction. Second, direct, rather than indirect, teaching of critical thinking 
has produced significant results. Third, there is little evidence to support or refute the 
efficacy of selected instructional methods. 

Measuring gains in critical thinking is difficult particularly since there is limited 
research to provide guidance and existing measures are fraught with problems. The 



literature suggested departing from current practice to obtain more accurate results. 
Recommendations included: use multiple measures of critical thinking; assess critical 
thinking skills frequently and over longer periods of time; employ alternative measures 
such as stimulated recall and concept mapping; take steps to ensure the assessments are 
valid, reliable, and fair; and develop subject-specific critical thinking tests. 
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Chapter 3 

METHODOLOGY 

The purpose of this study was to explore adult educators’ perceptions of topics 
related to critical thinking, particularly definitions of critical thinking, instructional 
methods, and methods of measuring growth in students’ thinking skills. The study used 
descriptive statistical methodology to summarize, organize, and simplify data to increase 
knowledge about adult educators’ perceptions of critical thinking. Descriptive research is 
dependent upon instrumentation for measurement (Gall, Borg, & Gall, 1996). This study 
used a self-report questionnaire to elicit responses from adult educators and used causal- 
comparative methods which Gall et al. describe as “a type of quantitative research that 
seeks to discover possible causes and effects by comparing individuals in which a 
phenomenon is present with individuals in whom the phenomenon is absent or present to 
a lesser degree” (p. 380). 

, This chapter describes the population and sample studied, instrumentation, the 
null hypotheses, and the analysis of data. 

Description of the Population 

The target population for this study was adult educators who currently teach or 
previously taught adult education courses in institutions within the United States offering 
graduate degrees in adult education. Peterson’s Guide to Graduate Programs in Business, 
Education. Health. Information Studies. Law and Social Work (1997), hereafter referred 
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to as Peterson’s Guide , listed 83 U. S. colleges and universities offering graduate degrees 
in adult education. One questionnaire, addressed either to Director, Adult Education 
Program, or to a specific individual listed as the program director in Peterson’s Guide. 
was sent to each of the 83 programs. 

Additionally, mailing labels for the current membership (September, 1997) of the 
Commission of Professors of Adult Education, a subdivision of the American Association 
for Adult and Continuing Education, were obtained. Of the 220 members listed, only 
those who were located in communities where graduate programs in adult education were 
being offered (according to Peterson’s Guide . 1997) were included in the survey. 
Members who lived outside the United States were eliminated from the sample as were 
members whose listings clearly indicated they were not teaching adult education courses. 
After eliminating duplications between the two lists, another 72 names were added to the 
original 83, resulting in a sample size of n = 155. 

Sample 

The mailing to the sample included a request from the researcher’s advisor asking 
adult educators to respond to the questionnaire, a cover letter to adult educators (see 
Appendix A), the three-page questionnaire (see Appendix B), and a stamped, self- 
addressed envelope for returning the completed questionnaire. The mailing was sent on 
October 17, 1997, requesting responses by October 31, 1997. 
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Of the 78 questionnaires returned, 10 were not completed either because the 
institution no longer had a graduate degree program in adult education or because the 
addressee had moved and left no forwarding address. Usable questionnaires totaled 68 
(out of a possible 145 remaining) for a response rate of 47%. No follow up was 
attempted since a 45% response rate is considered adequate for survey research (Suskie, 
1992). Analyses for items with n < 68 reflect missing values. 

Teaching Load, Years of Teaching Experience, Academic Rank 

Over half of the respondents (51 .5%) reported teaching adult education classes 
100% of the time. Nearly two-thirds of the respondents (62.1%) reported teaching adult 
education courses three-fourths time to full time (see Table 5). 

Table 5 

Teaching Load, Frequency and Percent Distributions 



Time 

Teaching 


Frequency 


Percent 


0- 25% 


9 


13.6 


26- 50% 


6 


9.1 


51 - 75% 


10 


15.2 


76- 100% 


41 


62.1 


Totals: 


66 


100.0 
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Over two-thirds of the respondents (67.2%) have taught adult education courses 
for ten years or more. Less than 12% of the respondents (1 1.9%) have three or fewer 
years of experience teaching adult education classes (see Table 6). 

Table 6 

Number of Years Teaching Experience. Frequency and 
Percent Distributions 



Years 


Frequency 


Percent 


m 

1 

o 


8 


11.9 


4-6 


8 


11.9 


7-9 


6 


8.9 


10 + 


45 


67.3 


Totals: 


67 


100.0 



Nearly half of the respondents (45.6%) were full professors. Less than 10% of the 
respondents (7.3%) were instructors. One respondent provided a title (i.e., adjunct 
professor) other than those listed on the questionnaire (see Table 7). 
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Table 7 

Academic Rank. Frequency and Percent Distributions 



Rank 


Frequency 


Percent 


Instructor 


5 


7.3 


Assistant Professor 


11 


16.2 


Associate Professor 


20 


29.5 


Full Professor 


31 


45.6 


Other 


_1 


1.4 


Totals: 


68 


100.0 



Instrumentation 

For this study, a three-page instrument (see Appendix B) was developed to 
address the research questions. A panel of three experts (an adult educator, a 
psychologist with expertise in survey research, and a mass communications professor 
who had conducted survey research of journalism and mass communications instructors 
on critical thinking skills) was invited to review a draft of the instrument. Two 
responded, and their recommendations were incorporated into the final draft of the 
instrument. 
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The questionnaire asked for responses in six areas. The first section requested 
demographic data about teaching load (i.e., percentage of time spent teaching adult 
education courses), number of years teaching experience in adult education, and 
academic rank. 

The second section included three questions about critical thinking. To help 
answer the first research question (Is critical thinking being taught in adult education 
classes?), respondents were asked if they included critical thinking in their adult 
education curriculum. Those who responded affirmatively were asked to what extent 
they taught critical thinking skills and whether they used a direct or indirect approach to 
teaching critical thinking. 

To obtain definitions of critical thinking (the second research question), the third 
section of the questionnaire asked respondents to indicate the extent to which they agreed 
or disagreed with eight statements about characteristics and definitions of critical 
thinking borrowed from adult educators and theorists, using a scale including 1 = 
“Strongly Agree”, 2 = “Agree”, 3 = “Uncertain”, 4 = “Disagree”, and 5 = “Strongly 
Disagree”. 

The fourth section of the questionnaire was designed to investigate which 
instructional methods were being used to teach critical thinking skills (research question 
three). Respondents were asked to indicate the extent to which they used 18 specified 
instructional methods, using a scale including 1 = “Do Not Use”, 2 = “Occasionally Use”, 
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and 3 = “Frequently Use”. At the end of this section, space was provided for respondents 
to list additional instructional methods and rank the extent to which each was used. 

Section five was designed to help answer the fourth research question (How do 
adult educators measure gains in students’ critical thinking?). This section contained six 
statements about evaluating students’ critical thinking. Items used the same three-point 
scale described for section four (above). At the end of this section, space was provided to 
allow respondents to list and rank additional evaluation methods. 

The final section of the questionnaire asked respondents to mark their responses 
on a five-point scale indicating the extent to which they agreed or disagreed with 17 
statements about a variety of topics related to adult education and critical thinking. This 
section was designed to answer the fifth research question (What are the perceptions of 
adult educators on topics related to teaching critical thinking?). 

While no single questionnaire on critical thinking was found to fulfill the purpose 
of this study, the instrument used by Ruminski and Hanks (1995) was acquired prior to 
development of questionnaire items (as described above). Permission was granted to use 
all or parts of the instrument (H. J. Ruminski, personal correspondence, August 27, 

1997). Although the purpose, goals, and target population of their study were different 
from those of the present study, their instrument was used to provide guidance in 
developing five items used in the questionnaire for this study. For the second section of 
the questionnaire related to instructing students on how to think critically, two items from 
the Ruminski and Hanks instrument were modified and used. For the final section of the 



O 

ERIC 



83 



74 



questionnaire related to adult educators’ perceptions of teaching criticalthinking, one 
item from the Ruminski and Hanks instrument was used, and two other items were 
modified and used. 

Research Questions 

The following section restates the research questions and lists related statistical 
hypotheses. The level of significance at which statistical hypotheses were tested was set 
at p = .05. This value was adjusted to control for Type I error rate. 

Research Question 1 : To what extent is critical thinking being taught in adult 

education classes? 

Responses related to this question were summarized using frequencies and 
percents. No statistical hypotheses were tested for this research question. 

Research Question 2 : How do adult educators define critical thinking? 

Analysis consisted of comparing adult educators’ responses with practices 
discussed in the review of literature. No statistical hypotheses were tested for this 
research question. 

Research Question 3 : Do adult educators have a preference in how critical 
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Although there were a number of items on the questionnaire about 
instructional methods, selected items were analyzed because of their interest to 
the investigator. 

Ho': There will be no difference in the proportion of more experienced 

adult educators and less experienced adult educators in the use of 
open-ended inquiry. 

Ho 2 : There will be no difference in the proportion of adult educators 

who agree critical thinking “involves higher order thinking, 
problem solving, and metacognition” and those who disagree with 
this definition of critical thinking in the use of reflection. 

Ho 3 : There will be no difference in the proportion of adult educators 

who use a direct approach and adult educators who use an indirect 
approach to teaching critical thinking skills in the use of small 
groups. 

Ho 4 : There will be no difference in the proportion of adult educators 

with the rank of full professor and adult educators with a lesser 
rank in the use of lecture. 

Ho 5 : There will be no difference in the proportion of adult educators 

who teach critical thinking all the time and adult educators who 
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seldom or occasionally teach critical thinking in the use of case 
studies. 

Ho 6 : There will be no difference in the proportion of adult educators 

who view critical thinking as a “behavioral skill” and adult 
educators who view critical thinking as a “cognitive skill” in the 
use of drill and practice. 

Ho 7 : There will be no difference in the proportion of adult educators 

who agree critical thinking is “reasonable reflective thinking” 
(Ennis, 1985, p. 45) and those who disagree with this definition in 
the use of logs/joumals. 

Research Question 4 : How do adult educators measure gains in students’ 

critical thinking? 

Ho 8 : There will be no difference in the proportion of more experienced 

and less experienced adult educators in the use of in-class 
discussion to evaluate students’ critical thinking. 

Ho 9 : There will be no difference in the proportion of adult educators 

who agree critical thinking includes “attitudes and values” and 
adult educators who disagree with this definition of critical 
thinking in the use of essay tests. 
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Ho 10 : There will be no difference in the proportion of adult educators 

who view critical thinking as a “behavioral skill” and adult 
educators who view critical thinking as a “cognitive skill” in the 
use of subjective observations. 

Ho' 1 : There will be no difference in the proportion of adult educators 

who agree critical thinking includes “identifying and challenging 
assumptions, and exploring and imagining alternatives” 
(Brookfield, 1987, p. 15) and adult educators who disagree with 
this definition of critical thinking in the use of interviews with 
students. 

Ho' 2 : There will be no difference in the proportion of adult educators 

who frequently use lecture as an instructional method and adult 

educators who occasionally use or do not use lecture in the use of 

) 

standardized tests. 

Research Question 5 : To what extent do adult educators agree or disagree on 

topics related to teaching critical thinking? 

Analysis consisted of comparing adult educators’ responses with practices 
discussed in the review of literature. No statistical hypotheses were tested for this 
research question. 
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Analysis of Data 

Data collected were coded and prepared for statistical analysis. JMP (SAS 
Institute, 1989-1996) statistical software was utilized to process quantitative data. Data 
were summarized using frequencies, percents, means, and standard deviations. Some 
tests for statistical significance were carried out. 
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Chapter 4 

\ 

ANALYSIS OF DATA 
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Introduction 

The purpose of this study was to investigate the perceptions of adult educators on 
topics related to defining, teaching, and evaluating critical thinking skills. This chapter 
presents data analysis results. Chi-square tests were used to test for statistical 
significance. Each research question is examined individually. 

Research question 1 dealt with teaching critical thinking skills. Respondents were 
asked if they included critical thinking in their curriculum. If they responded “yes”, they 
were asked to what extent they taught critical thinking skills: “seldom”, “occasionally”, 
“frequently”, or “all the time”. Respondents who instructed students on how to think 
critically were also asked if they “offer specific courses or units within courses on critical 
thinking skills (direct approach)” or if they “integrate critical thinking into subject-matter 
units (indirect approach)”. Results for research question 1 are reported using frequencies 
and percents. No tests of statistical significance were carried out. 

Research Question 1 : To what extent is critical thinking being taught in adult 

education classes? 

A vast majority of respondents (89.7%) indicated they included critical thinking 
in their adult education curriculum. Of those respondents, nearly half (46.8%) reported 
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they taught critical thinking skills “frequently” while another large group (40.3%) said 
they taught critical thinking “all the time” (see Table 8). 

Table 8 

Extent of Teaching Critical Thinking. Frequency and 
Percent Distributions 



Extent 


Frequency 


Percent 


Seldom 


0 


0.0 


Occasionally 


8 


12.9 


Frequently 


29 


46.8 


All the time 


25 


40.3 


Totals: 


62 


100.0 



A large majority of respondents (88.9%) reported integrating critical thinking 
skills into subject-matter units; i.e., they used the indirect approach to teaching critical 
thinking. Only one respondent reported using the direct approach to teaching critical 
thinking skills. 

Nearly 10% of respondents (9.5%) indicated they used both the direct and indirect 
methods of instructing students on how to think critically (see Table 9). 
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Table 9 



~ ~ ^ •> Zi 

Percent Distributions 


Method 


Frequency 


Percent 


Direct 


1 


1.6 


Indirect 


56 


88.9 


Both direct and indirect 


_6 


9.5 


Totals: 


63 


100.0 



Research question 2 dealt with definitions of critical thinking. Respondents were 
asked to indicate the extent to which they agreed or disagreed with eight statements 
related to characteristics and definitions of critical thinking. 

Results for research question 2 are reported using frequencies and percents. No 
tests of statistical significance were carried out. 

Research Question 2 : How do adult educators define critical thinking? 

There was little variation among responses by participants responding to 

» 

definitions of critical thinking. Most respondents (98.5%) agreed or strongly agreed with 
• Stephen Brookfield’s (1987) definition that critical thinking includes “identifying and 
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challenging assumptions, and exploring and imagining alternatives” (p. 15). Over 90 
percent of respondents (94.1%) agreed or strongly agreed with the statement that 
“Critical thinking involves higher order thinking, problem solving, and metacognition”. 
The same number of respondents (94.1%) agreed or strongly agreed with the statement 
that “Critical thinking includes attitudes and values”. A large number of respondents 
(85.3%) agreed or strongly agreed with McPeck’s (1981) definition that critical thinking 
involves “a certain skepticism toward given statements, established norm or mode of 
doing things” (p. 6). Over 80% of respondents (83.3%) agreed or strongly agreed with 
Ennis’ (1985) definition that critical thinking includes “reasonable, reflective thinking 
that is focused on deciding what to believe or do” (p. 45). 

In general, respondents tended to disagree or strongly disagree with three 
specified definitions of critical thinking. Eighty percent of respondents disagreed or 
strongly disagreed with the statement “Critical thinking is a behavioral, not a cognitive 
skill”. Nearly 70% of respondents (68.8%) disagreed or strongly disagreed that “Critical 
thinking is a set of discrete skills or proficiencies rather than a process”. Approximately 
two-thirds of respondents (67.1%) disagreed or strongly disagreed with the statement 
“Critical thinking is a synonym for problem solving”. 

Appendix C contains a table with individual ratings of definitions of critical 
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Research question 3 dealt with instructional methods. Respondents were asked to 
indicate whether each of 18 specified instructional methods was “Not Used”, 
“Occasionally Used”, or “Frequently Used”. Since there were 18 tests for statistical 
significance, Type I error rate was controlled by setting the level of significance at 
.05/1 8 = .003. Of particular interest were responses to the following instructional 
methods: open-ended inquiry, reflection, small groups, lecture, case studies, drill and 
practice, and logs/joumals. 

Research Question 3 : Do adult educators have a preference in how critical 

v thinking is taught? 

There was little variation in the responses of adult educators in the present study 
to 1 8 pre-selected instructional methods. In general, respondents indicated they 
frequently used the following instructional methods: small groups (85.1%); discussion 
(79.1%); reflection (77.2%); and open-ended inquiry (52.4%). Respondents indicated 
they occasionally used: lecture (65.7%); role play (62.1%); demonstration (61.5%); 
simulation (57.6%); games (55.4%); critical incidents (53.7%); brainstorming (53.0%); 
case studies (52.2%); and debate/logic (51 .5%). On the other hand, respondents indicated 
they did not use: programmed instruction (89.4%); drill and practice (87.9%); and think 
aloud interviews (63.5%). 
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Ho 1 : 


There will be no difference in the proportion of more 
experienced (9 + years teaching experience) and less 
experienced (0-9 years teaching experience) adult 
educators in the use of open-ended inquiry. 


Results: 


When grouped by experience, there was no statistically 

•y 

significant difference (% (2) = .92, p > .003) in the use of 
open-ended inquiry. 


Ho 2 : 


There will be no difference in the proportion of adult 
educators who agree critical thinking “involves higher 
order thinking, problem solving, and metacognition” and 
those who disagree with this definition of critical thinking 
in the use of reflection. 


Results: 


There was no statistically significant difference 

(% 2 (2) = .38, p > .003) in the proportion of adult educators 


1 


who agreed critical thinking “involves higher order 
thinking, problem solving, and metacognition” and those 
who disagreed with this definition of critical thinking in the 
use of reflection. 


Ho 3 : 


There will be no difference in the proportion of adult 
educators who use a direct approach and adult educators 


0 
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Results: 



Ho 4 : 



Results: 



Ho 5 : 



Results: 



who use an indirect approach to teaching critical thinking 
skills in the use of small groups. 

There was no statistically significant difference 
(X 2 (4) = 4.28,p> .003) in the proportion of adult educators 
who used a direct approach and adult educators who used 
an indirect approach to teaching critical thinking skills in 
the use of small groups. 

There will be no difference in the proportion of adult 
educators with the rank of full professor and adult 
educators with a lesser rank in the use of lecture. 

When grouped by rank, there was no statistically 
significant difference (% 2 (2) = 1.90, p > .003) in the use of 
lecture. 

There will be no difference in the proportion of adult 
educators who teach critical thinking “all the time” and 
adult educators who “seldom” or “occasionally” teach 
critical thinking in the use of case studies. 

There was no statistically significant difference 
(X 2 (2) = 1 1 .41, p > .003) in the proportion of adult 
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educators who taught critical thinking “all the time” and 
adult educators who “seldom” or “occasionally” taught 
critical thinking in use of case studies. 

Ho 6 : There will be no difference in the proportion of adult 

educators who view critical thinking as a “behavioral skill” 
and adult educators who view critical thinking as a 
“cognitive skill” in the use of drill and practice. 

Results : Chi-square analysis could not be computed because of the 

response pattern. None of the respondents fit one of the 
groups. 

Ho 7 : There will be no difference in the proportion of adult 

educators who agree critical thinking is “reasonable 
reflective thinking that is focused on deciding what to 
believe or do” (Ennis, 1985, p. 45) and those who disagree 
with this definition in the use of logs/joumals. 

Results : There was no statistically significant difference 

(X 2 (2)=1.27,p> .003) in the proportion of adult educators 
who agreed critical thinking is “reasonable reflective 
thinking that is focused on deciding what to believe or do” 
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(Ennis, 1985, p. 45) and those who disagreed with this 
definition in the use of logs/joumals. 

Appendix D contains a table summarizing frequency and percent results for 
instructional methods. 

The instructional methods section of the questionnaire concluded with an open- 
ended question asking respondents to list additional methods used for teaching critical 
thinking. Ten individuals contributed responses; some provided more than one response. 
Two respondents listed “readings”. All other responses were listed only once and 
included methods such as autobiographies, real world projects, experiential learning, 
structured interviews, instrumentation, and contracts. 

Appendix E provides a complete list of responses to the open-ended question 
related to instructional methods. 

Research question 4 dealt with methods of evaluating students’ critical thinking. 
Respondents were asked to indicate whether each of six specified evaluation methods 
was “Not Used”, “Occasionally Used”, or “Frequently Used”. Since there were six tests 
for statistical significance, Type I error rate was controlled by setting the level of 
significance at .05/6 = .008. 

Of particular interest were in-class discussion, essay tests, subjective 
observations, interviews with students, and standardized tests. These topics led to the 
formation of statistical hypotheses. 
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Research Question 4 : How do adult educators measure gains in students’ 

critical thinking? 

Nearly all respondents (97.1%) stated they frequently or occasionally used in- 
class discussion to evaluate growth in critical thinking. A large majority of respondents 
(8 1 .6%) used interviews with students, and an even greater number of respondents 
(86.6%) used subjective observations to measure gains in critical thinking. Nearly three- 
fourths of the respondents (73.8%) reported using essay tests occasionally or frequently 
to evaluate student’s critical thinking. Under half of the respondents (45.3%) 
occasionally or frequently used teacher-made tests to evaluate critical thinking. None of 
the respondents (0%) used standardized tests to evaluate students’ increases in critical 
thinking. 




There will be no difference in the proportion of more 



experienced and less experienced adult educators in the use 



of in-class discussion to evaluate students’ critical thinking. 



Results : When grouped by experience, there was no statistically 

significant difference (% 2 (2) = 1 .53, p > .008) in the use of 



in-class discussion as an evaluation technique. 




There will be no difference in the proportion of adult 



educators who agree critical thinking includes “attitudes 
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and values” and adult educators who disagree with this 
definition of critical thinking in the Use of essay tests. 

Results : There was no statistically significant difference 

(X (2) = 1 .37, p > .008) in the proportion of adult educators 
who agreed critical thinking includes “attitudes and values” 
and adult educators who disagreed with this definition of 
critical thinking in the use of essay tests. 

Ho 10 : There will be no difference in the proportion of adult 

educators who view critical thinking as a “behavioral skill” 
and adult educators who view critical thinking as a 
“cognitive skill” in the use of subjective observations. 

Results : Chi-square analysis could not be computed because of the 

nature of the data. 

Ho' 1 : There will be no difference in the proportion of adult 

educators who agree critical thinking includes “identifying 
and challenging assumptions, and exploring and imagining 
alternatives” (Brookfield, 1987, p. 15) and adult educators 
who disagree with this definition of critical thinking in the 
use of interviews with students. 
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Results : There was no statistically significant difference 

(X 2 (2) = 2.28, p > .008) in the proportion of adult educators 
who agreed critical thinking includes “identifying and 
challenging assumptions, and exploring and imagining 
alternatives” (Brookfield, 1987, p. 15) and adult educators 
who disagreed with this definition of critical thinking in the 
use of interviews with students. 

Ho : There will be no difference in the proportion of adult 

educators who “frequently use” lecture as an instructional 
method and adult educators who “occasionally use” or 
“never use” lecture as an instructional method in the use of 
standardized tests. 

Results : Chi-square analysis could not be computed because of the 

nature of the data. 

Appendix F contains a table with frequency and percent results for methods of 
evaluating critical thinking skills. 

The evaluation methods section of the questionnaire concluded with an open- 
ended question asking respondents to list additional methods used for measuring 
students’ gains in critical thinking. Of 68 respondents, 36 (53%) submitted responses 
with some providing more than one response. The most popular response was papers. 
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Eleven individuals specifically listed papers as an evaluation method used although there 
was variation in the types of papers listed (e.g., reflection papers, reaction papers, 
research papers, position papers, critical papers). Other forms of writing were also listed. 
For example, three respondents listed essays; five listed joumals/joumaling. Nine 
respondents named projects, either individual or team, while seven listed portfolios. 

Other responses included evaluation methods such as case studies, self-assessment, peer 
evaluation, and literature reviews. 

Appendix G provides a complete list of responses to the open-ended question 
related to evaluation methods. 

Research question 5 dealt with adult educators’ opinions toward teaching critical 
thinking. Respondents were asked to indicate the extent to which they agreed or 
disagreed with 17 statements related to teaching critical thinking, using a five-point scale. 
Items of interest included the perceptions of adult educators regarding: the goal of adult 
education, the goal of critical thinking, the efficacy of teaching critical thinking, the 
clarity of definition of critical thinking, the value placed on critical thinking by students, 
and the efficacy of using standardized tests to measure gains in critical thinking. 

Results for research question 5 are reported using frequencies and percents. No 
tests of statistical significance were carried out. 
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Research Question 5 : To what extent do adult educators agree or disagree on 
topics related to teaching adult education and critical thinking? 

All respondents (100%) strongly agreed or agreed that one of the goals of adult 
education should be to develop students’ critical thinking skills. Over 90% of 
respondents (94.1%) strongly agreed or agreed that the goal of critical thinking should be 
to produce students who are better thinkers in real-world contexts. The remaining 6% 
reported they were uncertain. 

Nearly two-thirds (65.7%) of respondents disagreed or strongly disagreed that 
most adult educators have a clear idea of what critical thinking is. Over half the 
respondents (51.5%) disagreed or strongly disagreed that instruction in critical thinking 
should be based on some conceptual definition that is comprehensive and measurable. 

Nearly 80% of respondents (79%) strongly agreed or agreed that critical thinking 
is a major objective in their curricula. Nearly 60% of respondents (58.2%) strongly 
agreed or agreed there is considerable evidence that critical thinking skills can be taught 
in adult education classes. Another 31.3% of respondents indicated they were uncertain 
about this statement. More than half the respondents (54.6%) disagreed or strongly 
disagreed that critical thinking skills can be facilitated but not taught. Another 27.3% 
strongly agreed or agreed with the statement while the remaining 18.2% were uncertain. 
Over 70% of respondents (70.2%) disagreed or strongly disagreed that critical thinking is 
best taught by direct, systematic instruction. 
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Over 60% of respondents (61.1 %) strongly agreed or agreed that many students in 
adult education classes lack fundamental critical thinking skills. While 50.8% of 
respondents disagreed or strongly disagreed that students learn to think critically without 
explicit instruction, another 32.8% agreed with this statement. 

There was consensus among respondents regarding the use of standardized tests. 
Ninety-one percent disagreed or strongly disagreed that growth in critical thinking can 
best be measured with standardized tests; the remaining nine percent were uncertain. 

Appendix H contains a table with frequency and percent results for adult 
educators’ perceptions of topics related to teaching critical thinking. 

Summary 

This chapter reported results of data analysis for this study. Research questions 1 , 
2, and 5 used descriptive results. Tests for statistical significance were conducted for 
hypotheses related to research questions 3 and 4. No statistically significant results were 
found. Chapter 5 contains a summary and discussion of the results, conclusions, and 
recommendations for future research. 





Chapter 5 



SUMMARY, DISCUSSION, AND RECOMMENDATIONS 
FOR FUTURE RESEARCH 

“The real act of discovery consists not in finding new lands 
but in seeing with new eyes. ” 

- Marcel Proust 

The purpose of this exploratory study was to examine adult educators’ 
perceptions of components of critical thinking, particularly definitions of critical 
thinking, instructional methods used for teaching critical thinking, and methods of 
measuring students’ growth in critical thinking. 

In the study, adult educators were defined as individuals who are currently 
teaching or previously have taught graduate-level courses in adult education in 
institutions granting graduate degrees in adult education. The study utilized a 
questionnaire, developed by the researcher, to obtain adult educators’ perceptions of 
critical thinking. 

The research questions that guided this study were: 

1 . To what extent is critical thinking being taught in adult education 
classes? 

2. How do adult educators define critical thinking? 

3. Do adult educators have a preference in how critical thinking is 



taught? 
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4. How do adult edqcators measure gains in students’ critical 
thinking? 

5. To what extent do adult educators agree or disagree on topics 
related to teaching critical thinking? 

Data related to research questions one, two, and five were summarized using 
frequencies and percents. Research questions three and four generated statistical 
hypotheses that were tested using the chi-square statistic. Discussion of the findings 
follows. 



Discussion of Findings 

Extent to Which Critical Thinking Is Being Taught 

Research question 1 dealt with the extent to which critical thinking skills were 

being taught in adult education classes. Results revealed adult educators believed one of 

the goals of adult education should be to develop students’ critical thinking skills. This is 

consistent with the adult education literature (e.g., Brookfield, 1987; Candy, 1991; 

0 

Garrison, 1991 ; Mezirow, 1981). Garrison claimed the most fundamental role of adult 
educators is to encourage and develop critical thinking because adult learners have the 
experience and intellectual and personal maturity to engage in critical thinking while pre- 
adults are not capable of or prepared for critical reflection. The findings of the present 
study also suggest adult educators agreed with Brookfield that “Learning to think 
critically is one of the most significant activities of adult life” (p. ix). 
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Respondents indicated they included critical thinking in their curriculum 
“frequently” or “all the time”. This suggests adult educators believe intensive programs 
are more effective than programs providing only periodic training in critical thinking and 
supports results of the meta-analysis by Bangert-Drowns and Bankert (1990). 

The literature suggests there is a growing body of research that supports the direct 
teaching of critical thinking skills (Chance, 1986; Halpem, 1993). Despite this research, 
a large majority (88.9%) of adult educators in this study reported integrating critical 
thinking skills into subject-matter units; i.e., they used the indirect approach to teach 
critical thinking. This preference for the indirect method may exist because adult 
educators lack training and experience in directly teaching critical thinking skills. 

Further, given the amount of curricular material to be covered as well as the constraints 
of time, adult educators may be unable to offer specific courses or units on critical 
thinking. 

A previous self-report study of journalism and mass communications faculty 
across the United States yielded nearly identical results. •Ruminski and Hanks (1995) 
wrote that 89% of respondents in their study reported using the indirect method of 
teaching critical thinking. However, the current study differed in two substantial ways 
from the Ruminski and Hanks study: level of education and discipline. The present 
study examined the perceptions of adult educators teaching graduate level courses in 
adult education while the study by Ruminski and Hanks investigated the perceptions of 
journalism and mass communications instructors teaching at the undergraduate level. 




106 



97 

Although previous studies of adult educators were not available, results of the present 
study may indicate that adult educators teach critical thinking in the same or similar ways 
to educators of other disciplines and at other levels. 

Defining Critical Thinking 

Participants generally agreed in their responses to eight pre-selected definitions of 
critical thinking. Because of this level of agreement on all definitions, it was impossible 
to identify one definition of critical thinking. However, it was possible to identify 
dimensions and characteristics of critical thinking which were upheld by respondents and 
supported in the literature. 

Respondents agreed or strongly agreed with the definitions of Brookfield (1987), 
Ennis (1985), and McPeck (1981). These definitions share several characteristics of 
critical thinking which were supported in the literature. For example, all three definitions 
described critical thinking as a process, not a product, affirming the conclusion of others 
(e.g., Darkenwald & Merriam, 1982; Dewey, 1933; Ennis, 1985; Facione, 1990; Paul, 
1990; Siegel, 1988). In the current study, adult educators indicated support for critical 
thinking as a process by disagreeing with the statement “Critical thinking is a set of 
discrete skills or proficiencies rather than a process”. 

Definitions of critical thinking by Brookfield (1987), Ennis (1985), and McPeck 
(1981) also denoted critical thinking as a cognitive rather than a behavioral skill. The 
literature consistently supported cognitive psychologists and their definitions of critical 
thinking (Dixson, 1991; Facione, 1990; Halpem, 1993). Adult educators in the present 
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study agreed with these experts by rejecting the statement “Critical thinking is a 
behavioral, not a cognitive skill”. 

By defining critical thinking as a cognitive skill, respondents were in concert with 

current theorists. Although behaviorism was popular for several years, Schrag (1992) 

summarized the changeover to cognitive psychology: 

From the 1950’s until the 1970’s, most educational research and development was 
dominated by the behaviorism school of psychology, which either denied the 
existence of mental processes or denied their accessibility to scientific study. The 
last decade has witnessed the ascendancy of cognitive approaches to psychology- 
approaches that are well suited to the experimental investigation of thinking and 
problem-solving processes. The cognitive revolution in psychology, in turn, has 
penetrated the training and outlook of educational research and development 
workers, (p. 255) 

Although not stated explicitly, the definitions of Brookfield (1987), Ennis (1985), 
and McPeck (1981) suggested critical thinking involves both skills and dispositions. The 
dispositions of suspended judgment and skepticism crossed all three definitions and were 
also apparent in the findings of the Delphi Project which recognized these related 
dispositions: flexibility in considering alternatives and opinions; prudence in suspending, 
making, or altering judgments; and willingness to reconsider and revise views where 
honest reflection suggests that change is warranted (Facione, 1990). By agreeing with 
Brookfield, Ennis, and McPeck, adult educators in the present study may have been 
voicing support for a definition of critical thinking that involves both skills and 
dispositions, and particularly the dispositions of skepticism and suspended judgment. 

Further evidence that adult educators believed critical thinking includes 
dispositions is found in their responses to the statement “Critical thinking includes 
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attitudes and values”. By agreeing with this definition, adult educators in the present 
study supported the literature. Ennis (1985), Facione (1990), and Paul (1990) identified 
attitudes and values which critical thinkers habitually exhibit, such as: flexibility, open- 
mindedness, fairness, clarity, precision, and willingness to consider different points of 
view. 

Adult educators in the present study rejected the statement “Critical thinking is a 
synonym for critical thinking” but agreed with the statement “Critical thinking involves 
higher order thinking, problem solving, and metacognition”. Their responses supported 
the literature (e.g., D’Angelo, 1971; Facione, 1990; Kurfiss, 1988; Schrag, 1992) which 
reported that critical thinking is broader than problem solving. Facione (1990) listed the 
six cognitive skills identified by the 46 critical thinking experts who participated in the 
Delphi Project. These six cognitive skills can also be grouped under the categories of 
higher order thinking, problem solving, and metacognition. For example, interpretation, 
analysis, evaluation, inference, and explanation are types of higher order thinking and 
problem-solving skills while self-regulation is a metacognitive skill. 

In summary, the review of literature indicated no single agreed-upon definition of 
critical thinking. Respondents’ perceptions paralleled the literature in that no uniform 
definition of critical thinking emerged. However, there was consensus regarding 
characteristics of critical thinking. Adult educators in the present study and scholars cited 
in the review of literature viewed critical thinking as a process (not a product) and a 
cognitive skill rather than a behavioral skill. Further, both the literature and the subjects 
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of this study agreed that critical thinking involves both cognitive skills and dispositions. 
Finally, both agreed that critical thinking is broader than problem solving and in fact 
involves higher order thinking, problem solving, and metacognition. 

Instructional Methods for Teaching Critical Thinking 

Instructional methods used by adult educators to teach critical thinking skills were 
also investigated. In general, respondents reported frequent use of small groups, 
discussion, reflection, and open-ended inquiry and occasional use of lecture, role play, 
demonstration, simulation, games, critical incidents, brainstorming, case studies, and 
debate/logic. It is possible that adult educators used a broad repertoire of methods 
because they recognize that no single strategy or method can be effective for all learners, 
all learning styles, or all learning goals. On the other hand, results indicated neither 
programmed instruction nor drill and practice were used to teach critical thinking. 

More experienced and less experienced adult educators were compared on their 
use of open-ended inquiry. Results indicated respondents used open-ended inquiry 
regardless of how many years of teaching experience. Respondents seemed to agree with 
others (Brookfield, 1987; Candy, 1991; Darkenwald & Merriam, 1982; Dixson, 1991; 
Garrison, 1992; Knowles, 1980; Mezirow, 1981) that the role of the adult educator is to 
facilitate learning rather than to teach adult learners. Facilitators view open-ended 
inquiry as an effective instructional method, a perception which is borne out by the 
studies of Kurfiss (1988) and Baker and Anderson (1983). 
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Reflection was described in the literature as one of the preferred methods of adult 
education (Darkenwald & Merriam, 1982). Further, ethnographic studies by Clarke 
(1995) and Gipe and Richards (1992) investigated the use of reflection as a method of 
enhancing critical thinking and concluded that reflection improves critical thinking. 
Respondents in the current study supported these results with nearly 80% reporting 
frequent use of reflection in their teaching of critical thinking. Respondents used 
reflection as an instructional method whether they agreed or disagreed with the definition 
that critical thinking “involves higher order thinking, problem solving, and 
metacognition”. 

As a method of teaching critical thinking, reflection may be widely used by adult 
educators because of its close association with higher order mental processes. Mezirow 
and Associates (1990) referred to reflection as “intellectual and affective activities in 
which individuals engage to explore their experiences in order to lead to new 
understandings and appreciation” (p. 5). By definition, reflection includes cognitive 
skills and dispositions which adult educators previously affirmed as components of 
critical thinking. Further, reflection requires examination of life experiences, a tenet 
which adult educators uphold. 

Participatory techniques, including small group work, were rated highly in the 
adult education literature (Brookfield, 1987; Darkenwald & Merriam, 1982; Garrison, 
1991; Knowles, 1980; Mezirow, 1981). In addition, the effectiveness of paired problem 
solving and cooperative learning settings was reported by Flowe and Warren (1989) and 
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Kurfiss (1988). Therefore, it was expected that adult educators in the current study 
would use small groups frequently when teaching critical thinking. Adult educators who 
used a direct approach to teaching critical thinking and adult educators who used an 
indirect approach were compared in the frequency of use of small groups. Nearly all the 
respondents used the indirect approach and also used small groups, resulting in no 
statistically significant difference. 

Further, the frequent use of small groups by respondents confirmed results of 
earlier studies on the efficacy of using small groups to teach critical thinking (Gokhale, 
1995; Howe & Warren, 1989; Karabenick & Collins-Eaglin, 1996; Smith, 1977). 

Literature related to lecture as an instructional method yielded mixed results. 
Allen (1995) found the lecture format, when followed by other instructional methods 
such as in-depth discussion of a case, was effective in improving critical thinking skills. 
On the other hand, McKeachie (1970) cited seven studies to demonstrate lecture classes 
are less effective than discussion classes in promoting and retaining higher level thinking. 

Although lecture as an instructional method has been denigrated by some 
educators, Brookfield (1990b) and Freire (1989) believed it merits attention in adult 
education. Describing his own experience as an adult educator, Brookfield wrote that 
early in his career he refused to give lectures based on his understanding that they were 
“domineering, authoritarian, and disrespectful of students’ dignity” (p. 25). In time, he 
came to believe that lectures were not only appropriate but essential for certain purposes. 
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As a result, he began using lectures occasionally and has written book chapters on 
lecturing effectively for and with adult learners. 

Results of the present study tend to support Brookfield’s (1990b) position. In 
general, adult educators, regardless of their rank, used lecture but only occasionally. The 
types of lecture (e.g., mini lecture, participatory lecture, and traditional lecture) and the 
purposes for which respondents used lecture were not revealed. Even though adult 
educators are committed to experiential learning, it is possible they used lecture for 
specific purposes such as to impart complex or technical information or concepts 
unfamiliar to the learners. However, it appears likely that respondents used lecture 
infrequently because they believe adult education classes should be learner-centered 
rather than teacher-centered. 

The case study, as an instructional technique, has become increasingly popular 
(Allen, 1995; Boyce, 1993; Zeakes, 1992). Because the case study approach is a learner- 
centered, problem-solving approach, it was expected that respondents would use the case 
study frequently. As predicted, a large majority (91.0%) reported frequent or occasional 
use of case studies. Whetherthey taught critical thinking “all the time” or “seldom” or 
“occasionally”, adult educators included the case study as an instructional method. 
Previous studies (e.g., Allen, Boyce, Zeakes) indicated the case study method 
significantly increased students’ ability to think critically. 

Adult educators may use case studies because this approach provides 
opportunities for richer understanding of issues than could be gained from other 
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instructional approaches such as the lecture. Further, case analysis actively engages 
students in learning (a highly valued principle of adult education) and by its very nature 
requires use of critical thinking skills such as problem identification and formulation of 
possible solutions. Additionally, case studies may be used by adult educators because 
they make use of other important adult education techniques such as small group 
discussion or role play. 

Adult educators who defined critical thinking as a behavioral skill and adult 
educators who defined critical thinking as a cognitive skill were compared in their use of 
drill and practice. Results indicated no respondents viewed critical thinking as a 
behavioral skill or used drill and practice. It is possible that adult educators did not use 
drill and practice to teach critical thinking skills because this method is incompatible with 
adult education principles which emphasize experiential, learner-centered approaches. 
Findings from the current study were consistent with the results of the Gokhale (1995) 
study which demonstrated that drill and practice is an ineffective instructional method for 
teaching critical thinking. 

Journal writing has become a frequently used technique for discovering the nature 
of the learning process itself (Candy, 1991) and as a transformative tool for adults 
(Mezirow & Associates, 1990). Thus, it was expected adult educators would utilize 
journals or logs for teaching critical thinking skills. Nearly 90% of respondents reported 
using logs/joumals frequently or occasionally. Whether or not they agreed critical 
thinking is “reasonable reflective thinking that is focused on deciding what to believe or 
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do”, respondents used logs/joumals as an instructional method for teaching critical 
thinking. 

It is possible adult educators used logs/joumals because this method is learner- 
centered, allowing adult learners to record and leam from their thoughts and feelings at 
the time they occur. Further, journal writing requires a high degree of critical reflection. 
Previously, it was noted that adult educators attempt to teach critical thinking through the 
use of reflection. Finally, with the recent emphasis on brain research, it is possible adult 
educators used the journal as a way to gain understanding of how adults leam. 

In summary, respondents demonstrated little variation in the instructional 
techniques used to teach critical thinking. In general, findings were consistent with adult 
education literature. However, it is important to note the paucity of research in this area 
and the lack of evidence to support or refute the efficacy of selected instructional 
methods. 

Methods Used to Evaluate Critical Thinking 

Methods of evaluating gains in critical thinking skills were also investigated. In 
general, respondents reported frequent use of in-class discussion and frequent or 
occasional use of essay tests, interviews with students, and subjective observations. 
Interestingly, these methods fit under the category of “naturalistic evaluation” 
(Brookfield, 1986, p. 272) which advocates the use of qualitative rather than quantitative 
modes of data collection. Naturalistic evaluation has relevance for adult educators 
because of its emphasis on allowing learners to help set the curriculum and determine 
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evaluation criteria. Results of the current study also indicated standardized tests were not 
used to evaluate growth in critical thinking skills. 

Respondents, regardless of length of teaching experience, used in-class discussion 
to evaluate students’ critical thinking. Adult educators may use this approach because it 
has several advantages over written evaluation techniques. For example, through 
discussion, the evaluator can probe to discover the thinking processes students use as well 
as the assumptions they make as they solve problems. The discussion method also allows 
the evaluator to measure students’ dispositions (e.g., open-mindedness and willingness to 
change positions when evidence warrants) to think critically. Another advantage of the 
discussion method is that it allows for frequent evaluation as recommended in the 
literature. Also, adult educators, as described previously, used discussion as an 
instructional method to teach critical thinking. It seems a logical extension to use 
discussion as an evaluation tool. 

Regardless of whether they agreed or disagreed that critical thinking includes 
“attitudes and values”, respondents used essay tests to evaluate growth in critical 
thinking. This finding supports studies of critical thinking (Browne, Haas, et al., 1978; 
Kerka, 1992; McMillan, 1987; Norris, 1988) that claimed essay tests were superior to 
multiple choice tests in measuring students’ thinking skills. Adult educators may use 
essay tests because they go beyond products (that is, correct answers) to reveal students’ 



thinking processes. 
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Respondents, whether they defined critical thinking as a behavioral or a cognitive 
skill, used subjective observations to measure gains in critical thinking. This finding 
supports Brookfield’s (1986) argument that informality and flexibility are valuable 
characteristics of adult education evaluative procedures. Furthermore, qualitative 
approaches to evaluation of critical thinking, particularly interviews and observational 
methods, are gaining credibility (Brookfield). 

Interviews with students was another evaluation technique used by respondents to 
measure gains in critical thinking skills. The premise that learners should be involved at 
all stages in the teaching-learning process, including evaluation, is one of the 
fundamental principles of adult education. Student interviews as an example of 
“participatory evaluation” (Brookfield, 1986, p. 276) places much of the responsibility 
for generating evaluative criteria on learners. Therefore, it seems reasonable that adult 
educators would utilize interviews as a means of assessing, with students, the 
development of thinking skills. 

Respondents did not use standardized tests to evaluate growth in critical thinking 
perhaps because they recognize the shortcomings of such tests. The literature (Browne et 
al., 1978; Dowling, 1990; Ennis, 1993; Halpem, 1993; McMillan, 1987; Norris, 1988) 
described several weaknesses of standardized tests. For example, standardized tests 
measure products, not processes of critical thinking. They are unable to measure 
affective dimensions of critical thinking. Furthermore, standardized tests are often 
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summative (final) evaluation whereas adult educators are committed to formative 
(frequent, ongoing) evaluation. 

When asked which evaluation methods they used beyond the six listed on the 
questionnaire, several respondents named papers, projects, portfolios, and journals. It 
may be that these methods will become part of the evaluative model for adult education 
which Brookfield (1986) recommended. His concern was that systematic evaluation of 
adult learning is infrequent due to the absence of an evaluative model that is grounded in 
the concepts, philosophies, and processes of adult learning. 

Unfortunately, there were no available studies in the adult education literature 
regarding the efficacy of selected evaluation methods as a basis of comparison. From 
this study, it is not known why adult educators used in-class discussion, essay tests, 
interviews with students, and subjective observations to measure students’ growth in 
critical thinking. Nor is it known why adult educators did not use standardized tests to 
measure gains in critical thinking. Further, it is not known how frequently adult 
educators measure changes in students’ critical thinking or if the instruments are valid, 
reliable, and fair. These are concerns which were raised in the undergraduate literature 
and merit attention in adult education as well. 

Teaching Critical Thinking 

The final research question examined adult educators’ perceptions of topics 
related to teaching critical thinking. Participants responded that teaching critical thinking 
should be one of the goals of adult education, that critical thinking can be facilitated (or 

O 

ERLC 



18 



109 



taught) in adult education, and that critical thinking is a major objective in their curricula. 
These results support beliefs of adult educators referenced in the literature (Brookfield, 
1987; Candy, 1991; Darkenwald & Merriam, 1982; Garrison, 1991; Mezirow & 
Associates, 1990). 

On the other hand, respondents claimed that adult educators do not have a clear 
notion of what critical thinking is. It appears adult educators, like others (e.g., cognitive 
psychologists, behaviorists, and philosophers) have been unable to come to consensus on 
a standard definition of the construct. Respondents were divided about whether 
instruction should be based on some conceptual definition of critical thinking that is 
comprehensive and measurable. The literature, however, tended to favor teaching critical 
thinking based on a conceptual definition (Facione, 1990; McMurray et al., 1989). 

Further, adult educators in this study disagreed that critical thinking is best taught by 
direct, systematic instruction which conflicts with the results of studies by Browne, Haas, 
et al. (1978); de Bono (1983); Penner (1995); and Statkiewicz and Allen (1983). 

However, these studies involved undergraduate students in disciplines other than adult 
education. Perhaps critical thinking has a different meaning for adult education. 

Adult educators in the current study reported that one of the goals of teaching 
critical thinking should be to produce students who are better thinkers in real-world 
contexts, a perception that is consistent with others (e.g., Dowling, 1989; Kerka, 1992). 
However, respondents noted many students in adult education classes lacked fundamental 
critical thinking skills which parallels earlier studies carried out with undergraduate 
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students. One of these studies cited substantial evidence that college students have 
serious deficiencies in their ability to reason (Kurfiss, 1988). 

Respondents were divided on whether students learn to think critically without 
explicit instruction. However, there is some evidence that critical thinking skills are 
improved with explicit instruction (Browne, Haas, et al., 1978; de Bono, 1983; Penner, 
1995; Statkiewicz & Allen, 1983). Again, it should be noted that few research studies 
were available and those that were reviewed were conducted at the undergraduate level in 
a variety of disciplines. 

Conceptual Framework of Critical Thinking 

One important aspect of this study dealt with the definition of critical thinking for 
adult education. Based on results, it appears that a conceptual framework of critical 
thinking can be visualized as a two-dimensional construct (see Figure 1 .) The proposed 
framework indicates critical thinking consists of cognitive skills and the cognitive and 
affective dispositions to use them (Brookfield, 1987; Ennis, 1987; Facione, 1990; Paul, 
1982; Siegel, 1988). 

According to the proposed framework, the first dimension of critical thinking is 
cognitive skills. Respondents in this study agreed with others (Brookfield, 1987; Dixson, 
1 9S>1 ; Facione, 1990; Halpem, 1993; Kerka, 1992; McKeachie et al., 1986) that critical 
thinking consists of cognitive rather than behavioral skills. 
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Figure 1 . Proposed Conceptual Framework of Critical Thinking 



Behaviorism is based on two primary assumptions. First, the focus of study is 
observable behavior rather than internal thought processes. Learning is manifested by a 
change in behavior. Second, the environment shapes one’s behaviors. In other words, 
what one learns is determined by the environment, not by the individual learner (Merriam 





& Caffarella, 1991). 



Adult educators in this study rejected the behaviorism approach in favor of the 

cognitive approach which views perception, insight, and meaning as the key contributors 

to knowledge and understanding. According to the cognitive approach to thinking, some 

conditions that promote the construction of meaning include: 

when students are able to link new information to prior knowledge; when students 
connect classroom learning to real world tasks and issues; when students actively 
engage with the content, questioning its premises and applying it to new examples 
and situations; and when students develop a repertoire of cognitive and 
metacognitive strategies. (Jones, 1992, p. 167) 

A chief difference between behaviorists and cognitive psychologists is locus of 
control over the learning activity. For behaviorists, control lies with the environment; for 
cognitive psychologists it lies with the individual learner (Merriam & Caffarella, 1991). 
Another difference is the goal of learning. Behaviorists believe the goal is a change in 
behavior; cognitive psychologists claim the goal is constructing meaning (Merriam & 
Caffarella, 1991). 

It seems logical that adult educators would support the cognitive approach 
because of their beliefs that (1) learning is within the adult learner’s control; and (2) the 
learner’s experiences are central to the learning process. 

The proposed conceptual framework of critical thinking in Figure 1 suggests four 
cognitive skills which help define critical thinking: higher order thinking skills, problem 
solving, metacognition, and reflective thinking (also referred to as reflective learning and 
reflection). Although critical thinking experts (e.g., Ennis, 1987; Facione, 1990) have 
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proposed a number of cognitive skills, respondents in this study agreed to the four listed 
above. 

/ 

Higher order thinking is a term often used as a synonym for critical thinking. In 
the proposed framework, however, higher order thinking skills together with problem 
solving skills are defined to include five of the six cognitive skills identified by the 
Delphi Project (Facione, 1990) as being at the core of critical thinking. These skills are 
interpretation, analysis, evaluation, inference, and explanation. 

Although there is some overlap between higher order thinking skills and problem 
solving, there are also some important distinctions. Problem solving is defined as the 
cognitive activity that occurs when an individual attempts to overcome a problem 
(Mayer, 1994). Like higher order thinking skills, problem solving is cognitive because it 
occurs internally, but it differs from higher order thinking in that it is “a common and 
pervasive type of thinking, directed toward a specific goal” (Mayer, p. 4723). In the 
literature, problem solving was promoted as a significant critical thinking skill (e.g., 
Bangert-Drowns & Bankert, 1990; Beyer, 1985; D’Angelo, 1971; Darkenwald & 
Merriam, 1982; de Bono, 1983; Dressel & Mayhew, 1954; Howe & Warren, 1989; 
Knowles, 1980; Kurfiss, 1988; McKeachie et al., 1986; McPeck, 1981; Meiss & Bates, 
1984). 

The third cognitive skill proposed in the conceptual framework is metacognition, 
defined as “awareness of one’s own cognitive processes” (Schrag, 1992, p. 255). The 
basic principles of metacognition demonstrate the connectedness of metacognition with 
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other cognitive skills and with the affective dispositions of critical thinking. The 
fundamental principles of metacognition may be summarized as follows: (1) Learning 
activities and processes, rather than learning outcomes, must be emphasized. (2) 

Students should be helped to become aware of their learning strategies, self-regulation 
skills, and the relationship of these strategies and skills to learning goals. (3) The 
interaction of cognitive, metacognitive, and affective components of learning is central to 
instruction. (4) Higher cognitive learning goals which require deeper cognitive 
processing should be emphasized (Simons, 1994). 

The importance of metacognition as a critical thinking skill was recognized by 
Karabenick and Collins-Eaglin (1996), Kurfiss (1988), McKeachie et al. (1986), and 
Schrag (1992). In addition, metacognition was identified as the sixth and final cognitive 
skill by the critical thinking experts involved in the Delphi Project (Facione, 1990). 

Called self-regulation by the Delphi panelists, this skill was further described as requiring 
self-examination and self-correction. 

Metacognition is included in the proposed framework because assisting adult 
learners in developing metacognitive skills enables them to negotiate and meet the 
demands of a wide variety of employment, social, and civic situations. This is one of the 
purposes of adult education and one of the goals of critical thinking. 

The fourth and final skill included in the conceptual framework is reflective 
thinking which is shared between both dimensions of critical thinking: cognitive skills 
and dispositions (see Figure 1). Boud, Keogh, and Walker (1985) pointed out that 
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reflection is “a generic term for those intellectual and affective activities in which 
individuals engage to explore their experiences in order to lead to new understandings 
and appreciation” (p. 3). 

The definition of reflective learning reveals its cognitive aspect: “the process of 
internally examining and exploring an issue of concern, triggered by an experience which 
creates and clarifies meaning in terms of self and which results in a changed conceptual 
perspective” (Boyd & Fales, 1983, p. 100). Candy (1991) added that there are many 
ways to reconcile new learning with existing views of knowledge. Reflection is one of 
those ways. The dispositional aspect of reflection is made evident by Harris (1990) who 
claimed reflection is “socially conditioned and affective in nature” (p. 1 13). 

Reflective thinking is included in the proposed conceptual framework of critical 

thinking because respondents affirmed the definition by Ennis (1985) who declared that 

« 

critical thinking includes “reasonable reflective thinking”. In addition, many critical 
thinking experts and researchers (e.g., Allen, 1995; Brookfield, 1987; Candy, 1991; 
Clarke, 1995; Darkenwald & Merriam, 1982; Dewey, 1933; Dowling, 1990; Facione, 
1990; Kerka, 1992; McPeck, 1981; Mezirow & Associates, 1990) have concluded that 
reflection is an essential component of critical thinking. 

As a sub-component of reflective learning, the proposed conceptual framework of 
critical thinking (see Figure' 1) depicts “identify and challenge assumptions” from 
Brookfield’s (1987) definition of critical thinking. Brookfield believed that critical 
thinking takes place “when people probe their habitual ways of thinking and acting for 
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their underlying assumptions” (p. 15). Schlossberg (1981) concurred, stating the 
outcome of reflection is “a change in assumptions about oneself and the world” requiring 
“a corresponding change in one’s behavior and relationships” (p. 5). 

While many additional cognitive skills have been associated with critical 
thinking, this conceptual framework is limited to four cognitive skills because these are 
the four adult educators in the study agreed upon. Also, as noted previously in this 
chapter, there is substantial evidence from previous studies to support their inclusion in 
the proposed framework. 

The second dimension of critical thinking proposed in the conceptual framework 
is dispositions. It was reported previously that several critical thinking experts (e.g., 
Brookfield, 1987; Candy, 1991; Ennis, 1985, 1987; Facione, 1990; Halpem, 1993; Paul, 
1990; Siegel, 1988) have insisted that critical thinking has both a cognitive and a 
dispositional dimension. In support of the dispositional dimension, Brookfield (1987) 
declared that emotions, feelings, and intuitions are central to critical thinking. D’Angelo 
(1971) and the Delphi panelists (Facione, 1990) are among those who cataloged various 
dispositions of critical thinking ranging from curiosity to flexibility and honesty. 

Respondents in this study agreed that critical thinking includes dispositions as 
well as values and attitudes. Dispositions can be subdivided between cognitive and 
affective. The Delphi Project (Facione, 1990) concluded that for every cognitive skill 
associated with critical thinking there is a corresponding cognitive disposition. Further, 
the experts participating in the Delphi Project identified affective dispositions of critical 
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thinking; i.e., the personal traits, habits of mind, or attitudes that characterize good 
critical thinkers. < 

Respondents agreed with McPeck (1981) that critical thinking involves a “certain 
skepticism toward given statements, established norm or mode of doing things”. 
Skepticism, as a major affective outcome of critical thinking, was described by 
Brookfield (1987) as the testing of an idea or practice against an individual’s experience— 
“being wary of uncritically accepting an innovation, change, or new perspective simply 
because it is new” (p. 2) and “a readiness to test the validity of claims made by others” (p. 
22 ). 

Further, Brookfield (1987) claimed that critical thinking involves “exploring and 
imagining alternatives” (p. 8), an activity which he felt leads to “reflective skepticism” 

(p. 9). Thus the conceptual framework exhibits “explore and imagine alternatives” as a 
sub-component of skepticism. 

The proposed framework depicted in Figure 1 is one possible way to 
conceptualize critical thinking. It is hoped adult educators will evaluate the merit of the 
framework and build upon it. 



Conclusions 

There is a paucity of research reports related to critical thinking in adult 
education. This exploratory study queried adult educators in the area of teaching critical 

thinking skills to determine current status and provide insight for future research. Since 
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critical thinking skills are considered essential for living and working successfully in a 
rapidly changing world, this study provides direction of efforts for more research in this 
crucial area of adult education. 

Lack of an agreed-upon definition of critical thinking can impede research efforts 
related to effective teaching and learning. Indeed, as McMurray et al. (1989) pointed out, 
“efforts to teach critical thinking presume the ability to diagnose needs and measure 
intervention effects, and measurement, in turn, presumes the ability to define the 
construct being measured” (pp. 1-2). 

Until a standard definition of critical thinking has been ascertained, adult 
educators will not be certain what to teach or how to teach it. Furthermore, adult 
educators will not be able to measure growth effectively. Knowledge of any construct, 
including critical thinking, can only come from rigorous evaluation results. 

Although no single agreed-upon definition of critical thinking emerged from this 
study, respondents indicated consensus on a number of characteristics of critical thinking: 

1 . Critical thinking is a process, not a set of discrete skills or proficiencies. 

2. Critical thinking is a cognitive, not a behavioral skill. 

3 . Critical thinking involves both cognitive skills and cognitive and affective 

dispositions. 

I 

4. Critical thinking includes attitudes and values. 

Based on these results, a framework of critical thinking is posited for 
consideration by the adult education community. Critical thinking is comprised of two 
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dimensions: (1) cognitive skills, and (2) dispositions. Cognitive skills consist of higher 
order thinking skills, problem solving, metacognition, and reflective thinking. 
Dispositions include both cognitive and affective dispositions. From this study, only one 
affective disposition was identified; that is, skepticism (McPeck, 1981) with a sub- 
disposition of exploring and imagining alternatives (Brookfield, 1987). It is possible to 
measure each of these indicators of critical thinking. 

In addition to the dimensions of critical thinking identified above, the current 
study yielded results related to instructional methods and evaluation. From the study, it 
appears adult educators use experiential and participatory methods when teaching critical 
thinking skills. This confirms adult education literature which emphasized learner- 
centered techniques that draw on students’ experiences. To evaluate growth in students’ 
critical thinking skills, adult educators in the current study used a variety of qualitative 
measures, but they did not use standardized tests. This differs from the results of other 
studies (e.g., Ruminski & Hanks, 1995) and may indicate a difference in measurement 
techniques by adult educators as compared to educators in other disciplines. 

Recommendations for Future Study 

Based on the results of this study, the following are suggestions for future 
research. 

1 . While the results of this study showed agreement among adult educators 
related to pre-selected definitions of critical thinking, no standard 
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definition of critical thinking emerged. More research should be 
conducted to develop or adapt a uniform and comprehensive definition of 
critical thinking by adult educators or more clearly delineate dimensions 
of critical thinking. Preliminary work was begun with this study. An 
agreed-upon definition, consistent with the philosophy of adult education, 
is needed to provide guidance regarding the curriculum, instructional 
methods, and evaluation techniques related to critical thinking. A related 
question is: Are those skills considered “critical” in adult education 
different from the skills considered “critical” in other fields? 

2. Once consensus has been reached on a definition of critical thinking, 
research could proceed to determine effective instructional methods used 
to teach critical thinking skills in adult education. This study investigated 
which instructional methods were used by adult educators but did not 
address the question of the effectiveness of these methods because 
effectiveness cannot be determined at this time. 

3. Results of this study indicated adult educators used a variety of qualitative 
approaches to evaluate growth in critical thinking skills. Again, once a 
workable definition of critical thinking has been established, research 
should investigate validity and reliability issues related to these qualitative 
approaches. Further, no standardized tests were used by participants. 
Research is needed to determine if there are reasons beyond the lack of 
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definition why adult educators do not use standardized tests or other 
quantitative measures to evaluate gains in students’ critical thinking skills. 
Finally, despite the problems with measuring critical thinking skills, do 
adult learners who show improvement in their ability to think critically 
maintain these gains over time? Longitudinal studies are needed. 

4. An additional research question merits further study: How does training 
in critical thinking in the field of adult education transfer to other 
disciplines, to employment settings, and to social/interpersonal situations? 

This exploratory investigation has revealed a strong need for further research that 
would benefit adult educators and adult learners. However, it seems to have generated 
more questions than answers. 
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Appendix A 

COVER LETTER TO ADULT EDUCATORS 



October 15, 1997 



Dear Adult Educator: 

I am a graduate student in the Adult Education Program at Drake University. As part of 
my graduate thesis, I am conducting a survey of adult educators who are currently 
teaching or have previously taught adult education classes at the graduate level. 

I am investigating several components of critical thinking skills in adult education, 
including definitions, instructional methods, and evaluation techniques. Whether or not 
you include critical thinking as an intentional part of your teaching or curriculum, 
your responses are very important to my investigation. 

The questionnaire will require 1 5 minutes of your time. A postage paid envelope has 
been included for your convenience. Please return the completed questionnaire by 
October 31. As an incentive for assisting with this study, I can offer you a copy of the 
results. To receive the results, please include your name and address in the space 
provided on the questionnaire. 

Participants in the survey will not be identified by name or other characteristics that may 
reveal their identity. 

Please call me (515-281-5108) or Dr. Charles Greenwood (515-271-2120), my advisor, if 
you have questions regarding the questionnaire. Thank you for your professional 
cooperation with my research efforts. I look forward to receiving your responses. 

Sincerely, 



Joann M. Vaske 

Enclosures: Questionnaire 

Stamped self-addressed envelope 




4 



X X i 



